How Small Companies Set Up AI Governance (The Short Answer)

A small company needs exactly three documents: an acceptable-use policy, a tool and data approval gate, and a lightweight risk register. That is the entire governance stack. Everything else, the sprawling enterprise frameworks with steering committees and maturity matrices, is overhead you cannot afford and will not enforce.

I am Mahmoud Zalt, an independent senior AI systems architect with 16 years of production software behind me since 2010. Governing a live workforce of autonomous agents at Sista AI, the company I founded, forced me to build the kind of guardrails this framework describes long before I wrote them down. As a Fractional AI Officer, I have set up governance models for small teams that are actually enforced because they fit on a single page and assign an owner to every rule. This article gives you that model in full.

Why Enterprise AI Governance Frameworks Fail Small Teams

ISO 42001, NIST AI RMF, and the EU AI Act compliance checklists were written for organizations with dedicated legal, compliance, security, and ethics teams. A 20-person company does not have those functions. When you hand a small team a 60-page framework, one of two things happens: nobody reads it, or one person reads it and writes a 40-page policy that nobody else reads.

The signal I watch for in small teams is the 'shelf test': if the policy document has not been opened in 90 days, it is not governance, it is liability theater. Real governance is three things working together.

• Clarity: every employee can answer 'what can I use AI for and what do I need approval for' in under 30 seconds.
• Ownership: one named person is responsible for approvals and the risk register, even if that is the founder or a part-time fractional role.
• Friction proportional to risk: low-risk uses (drafting internal emails) require no process; high-risk uses (AI touching customer PII, financial decisions, or code going to production) require a checklist and a sign-off.

Enterprise frameworks invert this. They add friction everywhere, which means engineers route around the process entirely.

Document 1: The Acceptable-Use Policy (AUP)

Your AUP answers one question: what is allowed, what is conditionally allowed, and what is prohibited. Keep it under 600 words. Every employee should be able to read it in five minutes and understand it without a lawyer.

The Three-Zone Structure

Divide uses into three zones.

What to Name Explicitly

Vague policies fail. Name the specific tools your team actually uses. 'ChatGPT and Claude are Green-zone tools for internal drafting but Yellow-zone for anything customer-facing. GitHub Copilot is Green-zone for feature code in your IDE, Yellow-zone for security-critical code paths, and Red-zone for code that handles authentication or payments without a dedicated security review.'

The One Rule That Prevents 80 Percent of Incidents

Add this sentence verbatim: 'Do not paste customer names, email addresses, API keys, passwords, internal financial data, or anything marked confidential into any AI tool that is not on the approved list.' This single rule, if followed, prevents the most common class of small-company AI data leak.

Document 2: The Tool and Data Approval Gate

The approval gate is a short checklist, not a committee. When any employee wants to use a new AI tool, especially one that will touch company or customer data, they fill in a five-field form and get a named approver to sign off. The goal is not to block tools. It is to ensure someone has looked at the data handling before the tool is in production use.

The Five-Field Approval Form

1. Tool name and vendor: what it is and who runs it.
2. Use case: one sentence on what the team will use it for.
3. Data classification: which data types will pass through it (internal only, customer data, PII, financial).
4. Data handling answer: does the vendor use your inputs to train models? Where is data stored? What is the retention period? (Check the vendor's data processing agreement or DPA, not the marketing page.)
5. Approved / conditionally approved / rejected + reason: the approver signs here with a date.

A Worked Example

Tool requested: Notion AI. Use case: summarize internal meeting notes and draft project specs. Data classification: internal only, no customer PII. Data handling: Notion's DPA states inputs are not used for model training, data is stored in the EU region, retention follows account settings. Decision: Approved, Green-zone, internal use. No customer data to pass through without a second review.

This takes 10 minutes to fill in. That is the right amount of friction for a Yellow-zone tool. For a Red-zone request, add a step: confirm with your legal or compliance contact (even if that is an external advisor) before approving.

Maintain a Living Approved-Tools List

Keep a simple table: tool name, approved date, zone, data types allowed, owner, review date. Review the list every six months. AI tools change their data handling policies, and a tool that was safe 12 months ago may have updated its terms to allow training on your inputs.

Document 3: The Lightweight AI Risk Register

The risk register is where you log what could go wrong, how likely it is, and what you are doing about it. For a small team, this does not need to be a spreadsheet with probability matrices. It needs five columns and an owner.

The Five Columns

• Risk description: a plain-English statement of the risk.
• Affected area: product, operations, legal, reputation, security.
• Likelihood (1-3): 1 = unlikely, 2 = possible, 3 = likely given current usage.
• Severity (1-3): 1 = recoverable with low cost, 2 = significant disruption, 3 = data breach, regulatory, or reputational damage.
• Mitigation: what you are doing now, even if it is just 'monitor' or 'review policy quarterly'.

The Risks You Almost Certainly Have Right Now

Without having seen your setup, here are the five risks that appear in nearly every small team I work with.

1. PII leakage via prompts. Employees pasting customer records or email threads into ChatGPT. Likelihood: 3. Severity: 3. Mitigation: AUP rule, approved-tools list, quarterly reminder.
2. Hallucinated output shipped without review. A developer or marketer accepts LLM output without checking it. Likelihood: 3. Severity: 2. Mitigation: Yellow-zone rule requiring human review before publish or merge.
3. Vendor terms change. An approved tool updates its DPA to allow training on customer data. Likelihood: 2. Severity: 3. Mitigation: six-month review cycle on the approved-tools list.
4. AI-generated code introduces a vulnerability. Copilot-suggested code with a SQL injection or insecure dependency lands in production. Likelihood: 2. Severity: 3. Mitigation: Yellow-zone rule for security-critical paths, mandatory code review for auth and payments code.
5. Over-reliance on a single vendor. Your team builds critical workflows around one LLM API with no fallback. Likelihood: 2. Severity: 2. Mitigation: design key workflows with model-agnostic abstraction, document the dependency.

What Small Teams Get Wrong (And How to Avoid It)

After setting up governance for multiple teams, the failure modes are predictable.

Writing Policy Without an Owner

If nobody owns the governance process, it decays in six months. Assign one person. In a team under 30, this is usually the CTO, the head of engineering, or a fractional AI advisor. Give them one hour per month to review the approved-tools list, update the risk register, and answer any Yellow or Red zone questions from the team. That is a realistic commitment.

Making Approval a Bottleneck

If the approval process is slow, engineers will bypass it. The five-field form should have a 48-hour turnaround commitment from the approver. If you miss that window, the requester gets a provisional approval with a 30-day review. Speed matters more than perfection at the SMB stage.

Treating Policy as a One-Time Document

The AI tool landscape changes quarterly. A policy written in January 2024 is already outdated in most of the vendor-specific details. Schedule a quarterly 30-minute review. Put it in the calendar now. Check whether any approved tools changed their DPAs. Check whether any new use cases have emerged that need a new zone classification.

Conflating Governance With Tool Restrictions

Governance is not about banning tools. It is about ensuring the team uses tools with eyes open. A policy that bans ChatGPT will be ignored. A policy that says 'ChatGPT is Green-zone for internal work and Yellow-zone for customer-facing work, and here is what that means' will be followed because it is useful, not threatening.

Making Governance Enforceable: Observability Without Surveillance

Governance is only real if you can observe it. For small teams, this does not mean logging every prompt. It means three lightweight signals.

1. A Shared AI Incident Log

Create a low-friction channel (a Slack thread, a Notion page, a GitHub discussion) where anyone can flag an AI-related incident or near-miss. 'I pasted a customer email into Claude by mistake' is a near-miss. 'Copilot suggested code that introduced a CVE and it shipped' is an incident. Log it, assign a remediation, close the loop. This is how you update the risk register with real data instead of theoretical risk.

2. A Monthly Usage Pulse

Ask the team one question per month: 'Did you use any AI tool this month that is not on the approved list?' This is not surveillance. It is discovery. Most shadow AI use is not malicious, it is people finding useful tools before the approval process has had a chance to catch up. The monthly pulse surfaces those tools so you can approve or restrict them formally.

3. Review Triggers for High-Risk Zones

Any Red-zone approval automatically triggers a 90-day review. Add it to the calendar at the time of approval. This ensures that a one-time exception does not become permanent undocumented infrastructure.

When You Need More: Scaling the Framework as You Grow

This three-document model is designed for teams of 5 to 50 people. As you grow past 50, a few things need to mature, but they build directly on this foundation rather than replacing it.

• At 50+ employees: formalize the approver role. Create a lightweight AI Review Board: the CTO or head of engineering, one person from legal or compliance if you have them, one senior engineer. The same five-field form, now with three approvers for Red-zone requests instead of one.
• When you take on regulated customers (health, finance, legal): the AUP needs a fourth zone, 'Regulated,' with explicit legal review required for any AI use touching that customer segment. Add the specific regulatory constraints (HIPAA data types, PCI-DSS card data) to the prohibited data list.
• When you deploy AI in your product (not just internally): governance expands to cover your product's AI behavior: evaluation benchmarks, hallucination rate targets, human-in-the-loop checkpoints, and a customer-facing disclosure. This is a separate workstream from internal-use governance, though the risk register feeds into both.

The three-document model scales because it is modular. You add zones and approvers without rewriting the core structure.

Frequently Asked Questions

How small is too small to need AI governance?

If you have at least two people using AI tools in their work, you need an acceptable-use policy. Even a one-page document that says what is allowed and what is not is governance. The failure mode is not having too little governance, it is having none at all and then discovering a PII leak after the fact.

Do we need a lawyer to write our AI policy?

No, not for the initial version. A lawyer should review it before it covers regulated data (health, financial, legal) or before you need to show it to enterprise customers. For most SMBs starting out, the founder or CTO can write the first version using the three-zone structure above. Get a legal review when the stakes rise, not before you have anything written.

What is the difference between an AI policy and AI governance?

The policy is one document inside governance. Governance is the system: the policy, the approval process, the risk register, the owner, and the review cadence. You need all five to have real governance. A policy document without an owner and a review schedule is a hope, not a system.

How do we handle employees using personal AI accounts at work?

Address it directly in the AUP. The practical rule: personal AI accounts (ChatGPT free tier, Claude.ai personal) are Green-zone for work that involves no company data, no customer data, and no code going to production. For anything else, use company-provisioned or company-approved accounts with known data handling terms. Do not ban personal accounts outright. It will not work and will erode trust in the policy.

What should we do when a vendor changes their data handling policy?

This is exactly why the six-month review cycle on your approved-tools list matters. When you spot a change, re-run the five-field approval form for that tool as if it were a new request. If the new terms move a tool from Green to Yellow or Yellow to Red, communicate the change to the team and update the list. If the change is severe enough (for example, a vendor now trains on your inputs by default), move the tool to the prohibited list until you have a satisfactory opt-out or a replacement.

Can AI governance help with AI vendor negotiations?

Yes, indirectly. When you have an approved-tools list with documented DPA requirements, you know exactly what to ask for in vendor conversations: data processing agreements, model training opt-outs, EU data residency, deletion timelines. Vendors are more likely to accommodate those requests in writing when the buyer arrives with a clear, specific list rather than vague security concerns.

Ready to Set Up AI Governance That Actually Works?

Most small teams are six months behind on AI governance, not because they are careless, but because they are building product and have no one whose job it is to own this. The three-document framework above is designed to be set up in a day and maintained in an hour per month. But 'set up in a day' only works if the person setting it up already knows which risks to prioritize and which enterprise complexity to skip.

That is exactly the kind of judgment I bring as a Fractional AI Officer. I have been doing this since before most enterprise AI governance frameworks existed. I know what scales and what dies on the shelf. If you want governance that your team will actually follow, and a technical partner who can build alongside it, get in touch.

Work with me as your Fractional AI Officer