The Volume Test: Does the Problem Occur Often Enough?

The first question I ask any team is: how many times per week does this task actually happen? If the answer is under 20, you almost certainly do not have an automation problem. You have an attention problem, and that is a management fix, not an engineering one.

A real example: a SaaS company wanted an AI agent to classify inbound support tickets and route them. Sounded sensible. When we counted, they had 30 tickets a week. One support rep, one shared inbox, one label column in a spreadsheet. The proposed agent build was estimated at 6 to 8 weeks of engineering and $200 to $400 per month in ongoing API costs. The spreadsheet took half an afternoon. I killed the agent project immediately.

A rough threshold that holds in practice:

• Under 50 occurrences per month: almost never worth an AI build. A human, a template, or a simple rule handles it.
• 50 to 500 per month: worth a structured automation (webhooks, low-code rules, a classifier with a confidence threshold and a human fallback). Consider AI only if the decision complexity is genuinely high.
• 500+ per month with real variance: now you have a candidate for an AI-assisted workflow. But even here, start with the simplest version first.

Unstable Rules Break Agents Faster Than They Break Humans

Agents encode your rules in prompts, retrieval indexes, and fine-tuned weights. When rules change, all three need to change in sync, and none of them tell you they are out of date.

Humans handle rule changes by reading a Slack message on Monday morning. An agent running on a prompt written in January will confidently apply January logic in October unless someone remembers to update it and then runs evals to confirm the update actually worked.

The failure modes are subtle. A pricing rule changes. The agent still quotes old prices, with high confidence, because the system prompt was not updated. Nobody notices for three weeks because the agent never says 'I am not sure.' It says 'your total is $340' and moves on.

Warning signals that your rules are too unstable for an agent:

• Rules live in a shared Google Doc that gets edited more than twice a month.
• Policy depends on jurisdiction, customer tier, or date ranges that shift regularly.
• You do not have a formal change management process for the rules themselves.
• The team cannot agree on the rule for a given edge case without a 30-minute conversation.

If any of these are true, build a rules engine or a configurable decision table first. Add AI on top later, once the rules are stable enough to test against.

Regulated, Irreversible, or High-Stakes Decisions Need Humans in the Loop

There is a category of decisions where being wrong is not just annoying but costly in ways that compound: lending decisions, medical triage, legal document generation, identity verification, HR terminations, and financial advice. AI can assist with all of these. AI should not be the final decision-maker for any of them, at least not yet and not without a documented human review step.

This is not about model capability. Modern LLMs are genuinely impressive at legal reasoning and medical literature synthesis. The issue is auditability, liability, and the asymmetry of errors. A wrong credit denial can violate fair lending law. A hallucinated drug interaction can harm someone. A confidently wrong contract clause can cost a client seven figures.

The standard I apply: if you cannot explain the decision trace to a regulator, a judge, or a patient in plain language, you should not let the AI make the final call alone. Human-in-the-loop is not a crutch. It is the architecture.

Concretely, this means:

• AI surfaces a recommendation with a confidence score and the top three supporting facts.
• A human reviews and approves or overrides before the decision is committed.
• The override is logged with a reason. You now have a feedback loop that improves the model over time.
• No agent takes an irreversible external action (send email, post transaction, update record) without an approval gate.

What a Simple Form, Rule, or Template Actually Beats

I keep a short mental list of things that routinely beat an AI agent on cost, reliability, and speed to production:

The test I use: if the logic can be expressed in a flowchart with under 10 nodes, write the flowchart and implement it directly. Reserve AI for problems where the input variance is genuinely high and the decision space cannot be enumerated.

The Hidden Costs Teams Forget When Scoping an AI Build

The API call is the smallest cost. The costs teams routinely undercount:

• Evaluation infrastructure: you need a test set, a scoring function, and a way to run both on every prompt change. This is not optional. Without evals, you are flying blind every time you update a prompt.
• Observability: structured logging of every LLM call, latency, token count, and output. If you cannot answer 'what did the model say to user X at 2pm yesterday,' you cannot debug or improve the system.
• Prompt maintenance: prompts drift. Models get updated. A prompt that worked on GPT-4o in March may behave differently in September. Someone owns this, or nobody does and the system quietly degrades.
• Guardrails: output validation, content filters, PII scrubbing, schema enforcement. Every production LLM call needs a layer that checks the output before it touches downstream systems.
• Fallback paths: what happens when the model returns low confidence, times out, or returns malformed output? If there is no fallback, the agent fails silently or loudly, and users see it.

A conservative estimate: for every $1 you spend on LLM API calls, budget $3 to $5 in engineering and infrastructure to make those calls production-safe. If that math does not work for your use case, the use case is not ready.

What Actually Makes a Good AI Automation Candidate

After ruling out the anti-patterns above, here is what a genuinely good candidate looks like. Use this as a checklist before committing to a build:

• High volume: the task happens hundreds of times per month and the volume is growing.
• High variance in input: the inputs are unstructured, free-form, or too diverse for a simple rule to cover.
• Low irreversibility: mistakes are catchable and correctable before they cause real harm. A misclassified support ticket is not a problem. A misfiled legal document is.
• Clear success metric: you can define what 'correct' looks like and measure it. No metric, no automation.
• Stable enough rules to write evals against: if you cannot write 20 test cases that define correct behavior, the problem is not well-defined enough to automate reliably.
• A human fallback exists: someone can handle the cases the agent gets wrong without the user experience breaking.

The classic good candidates in practice: document extraction from standard formats (invoices, resumes, forms), multilingual customer communication at scale, summarization of long structured content (contracts, reports), and intent classification feeding into a deterministic routing system.

Frequently Asked Questions

When is AI automation a bad idea for small businesses?

Almost always when volume is low. If your team handles a task fewer than 50 times per month, the engineering investment in an AI agent will never pay back. A template, a spreadsheet, or a simple intake form is faster to build, cheaper to run, and easier to change when requirements shift. Save AI automation for the repetitive high-volume work that is genuinely costing you hours per week.

What tasks should NOT be automated with AI?

Regulated decisions (lending, medical, legal), irreversible actions without a human approval gate, tasks where rules change frequently without a formal update process, any task that occurs fewer than a few dozen times per month, and anything where a template or simple rule already solves the problem. Also avoid automating tasks where you cannot define what 'correct' looks like, because you will have no way to know when the agent is wrong.

How do I know if my workflow needs AI or just a better process?

Start by mapping the workflow manually. If the bottleneck is unclear ownership or missing steps, that is a process problem. If the bottleneck is a human making a judgment call on unstructured input at high volume, that is a candidate for AI assistance. A good diagnostic: can you write a flowchart of the current process in under 20 minutes? If yes, implement the flowchart first. Add AI only if the flowchart fails to handle real-world variance.

Is AI automation worth it for low-volume use cases?

Rarely. The build cost, evaluation infrastructure, prompt maintenance, and observability tooling are largely fixed costs regardless of volume. At low volume, those fixed costs are almost never recovered. The exception is when the task is so specialized or cognitively demanding that even occasional instances justify the investment, such as a complex technical triage that would otherwise require a senior engineer every time.

What are the risks of automating too early with AI?

The main risks are silent degradation (the agent gets worse over time and nobody notices), compliance exposure (automated decisions in regulated areas without audit trails), and technical debt that is hard to unwind. Agents also tend to encode the assumptions of whoever wrote the original prompt. When the business changes, those encoded assumptions become liabilities. Teams that automate too early often end up with systems they are afraid to change because they do not understand what the agent is actually doing.

When should I use a rule-based system instead of an LLM?

Whenever the decision can be expressed as explicit logic, use explicit logic. Rule-based systems are deterministic, auditable, cheap to run, and easy to update. LLMs add value when the input is genuinely unstructured, when the decision space is too large to enumerate, or when natural language understanding is load-bearing. A good heuristic: if a new hire could learn the decision logic in a one-page document, it is a rule, not an AI problem.

Need a Straight Answer on Whether to Build?

Most teams I talk to come in wanting to automate something. Roughly half of them leave with a shorter scope than they started with, and a clearer, faster path to value. That is not a failure. That is good architecture.

If you are trying to decide whether an AI automation build makes sense for your workflow, I can give you a direct answer based on your actual volume, rules, risk profile, and existing tools. No pitch, no upsell, just a clear recommendation.

See what the AI automation work actually looks like, or get in touch directly to talk through your specific case.