A model sounds the same when it is wrong as when it is right. So an AI feature that works in your demo can quietly hand a real user a made-up answer in production. You cannot trust raw model output the way you trust code you wrote. This chapter gets your AI feature to fail safely instead of confidently.
7.5.1The model will be confidently wrong
A model has no sense of when it is wrong. It predicts fluent, plausible text, so a made-up answer reads exactly like a correct one.
Models hallucinate: they produce output that is fluent, plausible, and false. It is not a rare glitch you patch out, it is a property of how the model works.
So plan for wrong answers instead of hoping for right ones. Treat every response as an untrusted guess until something you control has checked it.
7.5.2Constrain the output and validate it
Free-form prose is hard to check by machine. So ask the model for structured output, a fixed shape like JSON, not a paragraph you have to interpret.
A shape you defined is a shape you can check. Validate every response against a schema with a validator like Zod, and reject anything that does not fit. It is the same discipline you apply to any external input.
Malformed output never flows downstream: a response that fails the check is dropped or retried, not passed to the user.
Once real users can reach the feature, wrong answers are not the only risk: some will try to make the model say something harmful or break its own rules, a jailbreak. The defense is a guardrail, a checkpoint that inspects what goes into and comes out of the model and blocks or rewrites anything unsafe. The easy win is running text through a moderation service instead of hand-writing filters; the input side, users smuggling instructions into their text, is the prompt-injection problem the security part covers.
7.5.3Evaluate it, do not eyeball it
You tweak a prompt, try it once, and the answer looks better. But one hand-check says nothing about the cases you did not try, and one fix often breaks another.
An eval is a set of example inputs paired with the results you expect, run against your feature after every change. It is testing aimed at a fuzzy feature. When you swap a model or edit a prompt, it flags the answers that got worse.
Start with five cases that matter, including ones the feature has gotten wrong before. Grow the set each time you find a new failure, the way a test suite grows.
An AI feature can also degrade silently over time as real-world inputs shift or the model behind it changes. That slow slide is drift, and it is why you keep the eval set running instead of checking once.
7.5.4Give it a timeout and a fallback
A model call is a network call to a service that can be slow or down, so it needs the same guardrails as any external call. Put a timeout on it, a limit on how long the call may run, so a slow response cannot hang your feature. Add a fallback, a safe answer for when the call times out, errors, or fails validation.
The fallback is whatever keeps the feature usable without the model: a cached answer, or an honest message that the smart version is unavailable. A bad model call should degrade the feature, never break it.
This prompt wraps one AI feature in all three defenses:
Do this now: paste the prompt with your riskiest AI feature, the one whose wrong answer a user would act on, and let your agent wrap it in a schema check, an eval, and a fallback.
