The Real Failure Mode: Organizational, Not Technical

When I audit a failed pilot, the post-mortem almost always reveals the same pattern. The team spent 80% of their effort on the model, the prompt, and the demo UI. They spent roughly zero effort on three things that actually gate production: a repeatable eval suite, a documented failure-mode inventory, and a named owner for the production decision.

The technology is rarely the problem. GPT-4o, Claude, Gemini, and open-weight models are all capable of powering most enterprise use cases right now. What kills pilots is organizational friction combined with an inability to prove the system is reliable enough to trust with real users, real data, or real decisions.

The Three Org Gaps That Kill Pilots

• No eval owner. Nobody is responsible for defining what 'good enough' looks like in measurable terms. Without that, every review meeting becomes a subjective debate and the pilot stalls indefinitely.
• No data governance sign-off. Production means real data. Legal and security need to approve data flows before go-live. Pilots that never loop in these stakeholders early get blocked at the finish line.
• No production owner. A pilot built by a data science team or an external consultant has no one to hand the pager to. If no internal engineer is assigned to own it in production, it will not ship.

Pick the First Use Case for Its Path to Production, Not Its Wow Factor

The single highest-leverage decision in any AI programme is the choice of first use case. Most teams pick the most impressive demo. I advise the opposite: pick the use case with the shortest, lowest-friction path from working prototype to production system.

A useful scoring rubric I use with clients has five dimensions, each scored 1-3:

A use case that scores 13 or above is a strong candidate for a first deployment. A use case that scores 8 or below, no matter how impressive the demo, is a pilot trap.

The classic pilot trap is the 'intelligent document understanding' demo. It looks extraordinary in a controlled setting. But the real documents have edge cases, the legal team needs to approve data handling, the blast radius of an error is high, and nobody wants to own the review queue when the model is wrong. It stalls for six months and gets quietly cancelled.

Evals First, Model Second

The eval suite is the production contract. Before you write a single line of prompt engineering, you need a set of test cases that define what the system must do, what it must never do, and how you will measure the difference. Without this, you cannot prove progress, you cannot prove regression, and you cannot make a credible case to a risk committee.

A minimal eval setup for a production-bound pilot looks like this:

• Golden set: 50-200 hand-labelled input/output pairs covering normal cases, edge cases, and known failure modes. These are your regression tests.
• LLM-as-judge: A secondary prompt that scores outputs on the dimensions that matter (accuracy, tone, groundedness, refusal correctness). Use a stronger model than the one you are deploying. Tune the judge against human scores until inter-rater agreement exceeds 85%.
• Hard constraint checks: Rule-based assertions that catch outputs that are categorically wrong regardless of subjective quality. For example: response must not contain PII, response must include a source citation, response must not recommend a specific product when the policy prohibits it.
• Latency and cost baselines: P50/P95 latency and cost per call. If production traffic is 10,000 calls per day, a $0.02 average cost is $200/day. Know this number before you demo to the CFO.

Run your eval suite on every prompt change, every model version bump, and every retrieval configuration change. Treat a regression in your golden set the same way you would treat a failing unit test: block the change until it is fixed.

Retrieval, Tool-Calling, and MCP: Where Pilots Actually Break

Most pilots that involve retrieval (RAG) or tool-calling (function calls, MCP) break at the integration layer, not the model layer. The model handles the reasoning fine. The failure is in the data pipeline, the tool contract, or the error handling around external calls.

Retrieval (RAG)

The two most common RAG failure modes I see in pilots are chunk boundary problems and retrieval precision collapse. Chunk boundary problems happen when a document is split mid-concept and the retrieved chunk lacks context. Fix this with overlapping chunks (10-15% overlap) and parent-document retrieval (retrieve the child chunk, return the parent). Retrieval precision collapse happens when the query embedding and the document embeddings are too dissimilar in distribution, usually because the documents were embedded with a different model or at a different time. Fix this by re-embedding all documents whenever you change the embedding model and by adding a reranker (cross-encoder) as a second-pass filter.

Tool-Calling and MCP

Tool-calling reliability in production requires three things that pilots routinely skip. First, every tool must have a strict input schema with validation, not just a description. The model will call tools with malformed arguments and your code must handle that gracefully. Second, every tool call must have a timeout and a fallback. A tool that hangs for 30 seconds will kill your p95 latency. Third, every tool call must be logged with its full input, output, latency, and error state. Without this log, debugging a production incident is nearly impossible.

MCP (Model Context Protocol) is increasingly the right abstraction for production tool use. It separates the tool definition from the orchestration layer, which makes it easier to audit, version, and swap implementations. If your pilot uses more than three external tools, MCP is worth the setup cost before you go to production.

Guardrails and Human-in-the-Loop Are Not Optional in Production

Every production AI system needs a layer between the model output and the real-world action. The shape of that layer depends on the blast radius of a mistake.

I use a simple three-tier model:

• Tier 1, auto-execute: The action is low-stakes and fully reversible. The model acts directly. Example: tagging a support ticket, summarising a document, generating a draft email that the user reviews before sending.
• Tier 2, human review queue: The action has moderate stakes or is hard to reverse. The model proposes; a human approves before execution. Example: scheduling a customer callback, updating a CRM field, generating an outbound communication.
• Tier 3, human-in-the-loop mandatory: The action is high-stakes, irreversible, or regulated. A human reviews the full context and the model's reasoning before any action is taken. Example: approving a financial transaction, changing account permissions, generating legal or medical advice.

Pilots that skip this tiering scheme and make everything Tier 1 for demo convenience get blocked by risk and compliance teams during production review. Build the tiering into the pilot from day one. It is far cheaper to design it in than to retrofit it under deadline pressure.

On guardrails specifically: use a defence-in-depth approach. Input guardrails (block prompt injection, PII in prompts, jailbreak attempts), output guardrails (check for PII in responses, check for policy violations, check groundedness against retrieved context), and rate limiting at the user and tenant level. The input and output checks do not need to be expensive. A fast, cheap classifier model (Haiku, flash-class) running in parallel with the main call adds less than 50ms and costs almost nothing at scale.

Observability and Cost: The Two Things That Kill Production AI Post-Launch

I have seen more production AI systems get pulled offline for cost overruns than for quality problems. And I have seen more production incidents take hours to diagnose because there was no observability. Both are entirely avoidable with upfront investment of maybe two to three days of engineering time.

Observability

Every LLM call in production must emit a structured log containing at minimum: trace ID, user/session ID, model ID and version, prompt token count, completion token count, latency (full round trip and time-to-first-token), retrieval hit/miss and retrieved chunk IDs if applicable, tool calls made and their outcomes, output text, and any guardrail flags triggered. Ship this to your observability stack (Datadog, Grafana, whatever you use) from day one. Set alerts on p95 latency exceeding your SLA, error rate exceeding 1%, and guardrail trigger rate exceeding a threshold that indicates prompt injection attempts.

Cost

Model inference cost scales linearly with traffic and token count. Before production, model your cost under three scenarios: current pilot traffic, 10x pilot traffic, and full production traffic at the stated target. If the 10x number is uncomfortable, you have a cost problem to solve before launch, not after. Common levers: switch to a smaller model for low-complexity tasks (the router pattern), cache deterministic or near-deterministic responses (semantic cache, exact cache), reduce context window by tightening retrieval precision, and use prompt caching where the provider supports it. A system that costs $0.004 per call at pilot scale can cost $4,000/day at production scale if traffic is 1 million calls/day. That number needs to be on the table before the go/no-go decision.

What Teams Consistently Get Wrong

After reviewing dozens of failed and stalled pilots, these are the patterns I see most often:

• They iterate on the prompt instead of the eval. Changing the prompt without a stable eval suite means you do not know if you are improving or regressing. The eval must come first.
• They use production data in the pilot without governance approval. This delays production sign-off by months because legal and security need to retroactively review data handling decisions that should have been made upfront.
• They build a monolithic agent when a pipeline would do. A single agent that does retrieval, reasoning, tool-calling, and output formatting in one pass is hard to debug, hard to eval, and brittle. Break it into stages. Eval each stage independently.
• They treat the model as a black box. Production AI requires you to understand where the model is likely to fail. Spend time on adversarial testing before you demo to stakeholders. Know your model's failure modes before they know yours.
• They skip the 'model wrong 5% of the time' conversation. Every stakeholder needs to understand before launch that the model will sometimes be wrong. The question is not 'is it perfect?' but 'is the error rate and the error cost acceptable relative to the baseline?' If you have not had that conversation explicitly, the pilot will fail at the first production incident.

Frequently Asked Questions

Why do AI proof-of-concept projects fail to scale to production?

The most common reason is that the proof-of-concept was optimised for demo performance rather than production reliability. It lacks evals, has no observability, was built on data that is not approved for production use, and has no internal owner assigned to maintain it. The gap from 'works in a notebook' to 'runs reliably at scale with a pager' is an engineering and organisational problem, not a model problem.

What is the biggest mistake companies make when starting an AI pilot?

Picking the use case for its impressiveness rather than its deployability. The right first use case is the one where you can write clear evals today, the data is already available and approved, the blast radius of an error is low, and you have a named engineer who will own it in production. Pick that one first, ship it, and build confidence. Then tackle the impressive use case with a team that knows how to ship.

How do you evaluate whether an AI pilot is ready for production?

Run your golden-set eval suite and confirm accuracy meets the agreed threshold. Confirm p95 latency and cost per call are within budget at projected production traffic. Confirm all data flows have been reviewed and approved by legal and security. Confirm guardrails and human-in-the-loop tiers are implemented and tested. Confirm observability is live and alerts are set. If all five are true, the pilot is ready. If any one is missing, it is not.

How long should an AI pilot take before it goes to production?

For a well-scoped first use case, six to twelve weeks is a reasonable target from kickoff to production deploy. Week one: use case scoping and eval design. Weeks two to four: prototype plus eval iteration. Weeks five to eight: integration, guardrails, observability, stakeholder review. Weeks nine to twelve: staged rollout, monitoring, hardening. Pilots that have been running for more than six months without a production date have almost always stalled on organisational gates, not technical ones.

Do I need a large team to run an AI pilot properly?

No. A two-person team, one engineer and one domain expert, can run a well-structured pilot if the use case is scoped tightly. What you cannot skip is the process: evals, stakeholder sign-off, observability, and a production owner. Those are process requirements, not headcount requirements. Adding more people to a pilot that lacks process does not help. It usually makes the stall worse.

What role does an AI consultant play in getting a pilot to production?

An independent AI consultant should do three things: help you pick the right first use case using a deployability framework, set up the eval infrastructure so the team can prove the system works, and navigate the organisational review path by identifying blockers early. The consultant should not be the production owner. That role must belong to someone internal. The goal of a good engagement is to leave the team capable of shipping the next pilot without external help.

Ready to Ship Your AI Pilot?

If your team has a pilot that looks good in demos but keeps stalling before production, the problem is almost certainly solvable. It requires honest use-case selection, a real eval suite, and a clear path through your organisation's review gates. None of that is exotic. It is just disciplined engineering applied to an AI system.

I work with teams as an independent AI consultant to do exactly this: scope the right first use case, build the eval infrastructure, set up observability and guardrails, and navigate the path to production. If you want a direct conversation about where your pilot is stalling, reach out on the contact page.

Talk to me about getting your AI pilot to production.