A model sounds the same when it is wrong as when it is right. So an AI feature that works in your demo can quietly hand a real user a made-up answer in production. You cannot trust raw model output the way you trust code you wrote. This chapter gets your AI feature to fail safely instead of confidently.
6.5.1The model will be confidently wrong
A model has no sense of when it is wrong. It predicts fluent, plausible text, so a made-up answer reads exactly like a correct one.
Models hallucinate: they produce output that is fluent, plausible, and false. It is not a rare glitch you patch out, it is a property of how the model works.
So plan for wrong answers instead of hoping for right ones. Treat every response as an untrusted guess until something you control has checked it.
6.5.2Constrain the output and validate it
Free-form prose is hard to check by machine. So ask the model for structured output, a fixed shape like JSON, not a paragraph you have to interpret.
A shape you defined is a shape you can check. Validate every response against a schema with a validator like Zod, and reject anything that does not fit. It is the same discipline you apply to any external input.
Malformed output never flows downstream: a response that fails the check is dropped or retried, not passed to the user.
Once real users can reach the feature, wrong answers are not the only risk: some will try to make the model say something harmful or break its own rules, a jailbreak. The defense is a guardrail, a checkpoint that inspects what goes into and comes out of the model and blocks or rewrites anything unsafe. The easy win is running text through a moderation service instead of hand-writing filters; the input side, users smuggling instructions into their text, is the prompt-injection problem the security part covers.
6.5.3Evaluate it, do not eyeball it
You tweak a prompt, try it once, and the answer looks better. But one hand-check says nothing about the cases you did not try, and one fix often breaks another.
An eval is a set of example inputs paired with the results you expect, run against your feature after every change. It is testing aimed at a fuzzy feature. When you swap a model or edit a prompt, it flags any regression, an answer that used to be right and got worse.
Start with five cases that matter, including ones the feature has gotten wrong before. Grow the set each time you find a new failure, the way a test suite grows.
An AI feature can also degrade silently over time as real-world inputs shift or the model behind it changes. That slow slide is drift, and it is why you keep the eval set running instead of checking once.
Drift is only visible if you kept the evidence, so record every model call from day one: what went in, what came back, which model answered, and whether validation failed. That is LLM tracing, and a tool built for it like Langfuse adds each call's tokens, latency, and cost. Every recorded failure is a free new eval case, and the Operate part turns these traces into alerts when quality slips.
6.5.4Every call spends your money, not theirs
The other way an AI feature hurts you has nothing to do with wrong answers. Every call your product makes is money you spend on behalf of one user, so your bill grows with how much they use it rather than with how many of them there are.
That makes one user enough to hurt you. A customer who leaves a tab open, a loop your agent wrote wrong, or somebody who notices your feature is free and expensive to run.
You are already recording every call for tracing. Record the user with it, and you can answer the two questions that matter: what does one user cost me, and is that less than they pay me.
Then give each account a quota, a hard limit on how much it can spend in a period, refused rather than warned about once it is hit. Free accounts get a small one, paying accounts get one sized to their plan.
Watch out: a limit on requests per minute is not a limit on cost. Ten expensive calls a minute, all day, is a bill you did not agree to. Cap the spend, not just the speed.
6.5.5Give it a timeout and a fallback
A model call is a network call to a service that can be slow or down, so it needs the same guardrails as any external call. Put a timeout on it, a limit on how long the call may run, so a slow response cannot hang your feature. Add a fallback, a safe answer for when the call times out, errors, or fails validation.
The fallback is whatever keeps the feature usable without the model: a cached answer, or an honest message that the smart version is unavailable. A bad model call should degrade the feature, never break it.
This prompt wraps one AI feature in all three defenses:
Do this now: paste the prompt with your riskiest AI feature, the one whose wrong answer a user would act on, and let your agent wrap it in a schema check, an eval, and a fallback.
