How to Control LLM Cost at Scale: The Short Answer

Controlling LLM cost at scale is an architecture decision, not a runtime tuning task. You need semantic caching to avoid re-running identical prompts, a model router that sends simple tasks to cheaper models, per-user and per-tenant hard limits enforced before the API call, and a cost-per-outcome metric visible in your dashboards so you catch drift before it compounds.

I am Mahmoud Zalt, an independent senior AI systems architect with 16+ years building production software. I designed Porto SAP, an architectural pattern for structuring large systems, and I bring that same structural thinking to Sista AI, the company I founded, where a year of running autonomous agents in production has made token economics a first-class design concern. I have designed and shipped AI systems that serve real traffic under real cost pressure. When clients come to me asking how to cut their LLM bill, the answer is almost always the same: the problem was baked in at design time. My AI Architecture advisory service exists specifically to fix that before (or after) it gets expensive. You can also read more about me here.

Why Token Spend Is an Architecture Problem

Most teams treat LLM cost like a cloud compute bill: let it run, watch the dashboard, optimize later. That works for EC2. It does not work for tokens, because token spend is proportional to every user action, every retry, every background job, every poorly scoped prompt. Three architectural mistakes cause almost every runaway bill I have seen in production.

Mistake 1: Sending Everything to the Frontier Model

GPT-4o and Claude Opus are remarkable models. They are also 20x to 50x more expensive per token than smaller models. Teams reach for the flagship by default, often because the prototype used it and nobody revisited the decision. A classification step that routes a customer query to the right handler does not need a frontier model. A summarization step over short structured data does not either. Sending every call to the top model is like renting a semi-truck to deliver a letter.

Mistake 2: No Caching Layer

In most production AI apps, 20 to 40 percent of prompts are semantically identical or near-identical. Without a cache, every one of those hits the API and burns tokens. Teams skip caching because 'AI responses should be fresh,' which is true for creative generation, not true for FAQ answers, classification outputs, or retrieval-augmented lookups on unchanged documents.

Mistake 3: No Per-User Limits in the Request Path

A single misbehaving user, a buggy client loop, or a viral moment can spike your monthly spend in hours. If the only thing standing between a rogue request loop and your credit card is your API provider's hard cap, you have no architecture, you have a prayer.

Build a Cost Model Before You Build the System

Before writing any prompt code, produce a cost model on a spreadsheet. This takes one hour and saves thousands of dollars. The inputs are: expected daily active users, average turns per session, average tokens per turn (prompt plus completion), and your model's per-million-token price. The output is a projected daily and monthly spend, plus a per-user-per-month cost.

That progression is real: a routing layer and a cache together can cut the bill by 85 percent on a typical workload, without degrading quality for the user. Do this math before the first sprint, not at the first billing alert.

Identifying Your Cost Drivers

Run a one-week sample of your actual prompts and measure: input tokens, output tokens, model used, endpoint, and user. Sort by total token spend descending. In every system I have audited, the top 10 percent of call sites produce 60 to 80 percent of the cost. That is where architecture changes pay off. Everything else is noise.

Model Routing: Send Easy Tasks to Cheap Models

A model router is a small classifier, a rule set, or a prompt that decides which model handles a given request. It is one of the highest-leverage architectural decisions in any LLM system. The logic is simple: cheap tasks go to cheap models; tasks that genuinely require reasoning, nuance, or long-context work go to the frontier.

A Worked Example: Customer Support Pipeline

Imagine a customer support assistant. Requests fall into roughly four buckets:

• Intent classification (is this a billing question, a bug report, or a refund request?): 50 tokens in, 10 tokens out. Route to GPT-4o-mini or Claude Haiku.
• FAQ match (does the question match a known answer in the knowledge base?): retrieval plus short completion. Route to GPT-4o-mini with a cached embedding lookup.
• Policy explanation (explain our refund policy in plain language): moderate context, predictable output. Route to GPT-4o-mini with a system prompt cache.
• Complex complaint (multi-turn, ambiguous, needs empathy and judgment): route to GPT-4o or Claude Sonnet.

In a real deployment I reviewed, the split was 15 percent complex, 85 percent easy. Routing correctly cut the per-conversation cost by 73 percent with no measurable drop in customer satisfaction scores.

How to Build the Router

Start simple. A rules-based classifier on intent labels, built with a small fine-tuned model or even keyword heuristics, is often enough. If you need ML routing, use a small embedding model (text-embedding-3-small, 0.02 cents per 1M tokens) to compare incoming requests against a labeled set, then threshold on cosine similarity. Do not build a complex routing system on day one. Start with two tiers, measure, then add tiers as the data justifies it.

Caching: The Highest-ROI Optimization in LLM Systems

There are three kinds of caching in LLM systems and each solves a different problem.

1. Exact Prompt Caching (Provider-Side)

Anthropic and OpenAI both offer prompt caching: if the prefix of your prompt matches a cached version, you pay 10 to 20 percent of the normal input token cost for that prefix. This is automatic if you structure your prompts correctly: put stable content (system prompt, large documents, tool definitions) at the top of the message array, and variable content (user message, dynamic context) at the bottom. A system with a 4,000-token system prompt and a 200-token user message can cache 95 percent of the input cost with zero code changes, just prompt structure. Do this first. It costs nothing to implement.

2. Semantic Response Caching (Application-Side)

Exact caching only helps when the prompt is byte-for-byte identical. Semantic caching goes further: embed the incoming query, find the nearest cached query by cosine similarity, and return the cached response if the similarity exceeds a threshold (typically 0.92 to 0.97 depending on how sensitive your domain is). Use Redis with a vector index (Redis Stack or a cheap pgvector setup) as the backing store. Set TTL based on how often the underlying data changes, not on a fixed duration.

Gotcha teams always hit: semantic caching on user-facing creative outputs degrades trust fast. 'Why did I get the same exact poem as my friend?' Cache classification and retrieval results, not creative or personalized completions.

3. Retrieval Result Caching

If you have a RAG pipeline, the retrieval step often dominates latency and contributes significantly to token cost (embedding + re-ranking). Cache retrieval results keyed on the normalized query plus document version hash. A document knowledge base that updates daily does not need fresh retrieval on every request, it needs fresh retrieval once per document update cycle. This is obvious in hindsight and almost always skipped in early architectures.

Per-User and Per-Tenant Limits: Enforcement in the Request Path

Rate limits and budget caps must be enforced before the LLM API call, not after. This sounds obvious but the implementation is frequently wrong: teams check limits in middleware, but the check and the API call are not atomic. Under concurrent traffic, a user can fire 10 parallel requests before any of them register against the counter.

The Correct Pattern

Use a Redis-based token bucket or sliding window counter, incremented atomically with a Lua script or Redis transaction, before issuing the API call. If the user exceeds the limit, return a 429 with a retry-after header. Do not call the API first and then try to refund the tokens: you cannot.

The limits to enforce at minimum:

• Per-user per-minute request limit: blocks runaway client loops and prevents a single session from monopolizing capacity.
• Per-user per-day token budget: derived from your cost model. A user spending 5x the median is either a power user (good, maybe upsell) or a bad actor or a bug (both bad).
• Per-tenant monthly cost cap: for B2B SaaS, each customer has a projected cost based on their plan. Alert at 80 percent, hard-stop at 110 percent.
• System-wide circuit breaker: if aggregate spend rate exceeds 2x the expected hourly rate, pause new requests, alert on-call, and investigate. A viral moment is exciting; a viral moment with no circuit breaker is a $40,000 surprise invoice.

Communicating Limits to Users

Show users how much of their quota they have used. Hide the token numbers (users do not understand them) and surface a percentage or a qualitative indicator. Users who can see 'you have used 80 percent of your daily AI credits' will self-throttle. Users who hit an unexplained wall will churn.

Cost Observability: You Cannot Control What You Cannot See

Token spend telemetry is not optional. Every LLM call in production must emit: timestamp, model, input tokens, output tokens, estimated cost (input tokens x input price + output tokens x output price), user ID or tenant ID, endpoint or feature, latency, and whether the response came from cache. Ship this on day one, before you need it.

The Metric That Actually Matters: Cost Per Outcome

Raw token counts tell you what you spent. Cost per outcome tells you whether you are spending efficiently. Define an outcome for your system: a completed task, a resolved support ticket, a successful code review, a converted lead. Track the token cost to produce that outcome over time. If cost per outcome is rising, something in your pipeline is degrading: prompts are growing, retries are increasing, the model is being called more times per task than before.

I have seen teams optimize token counts obsessively while missing that their retry rate had climbed from 2 percent to 18 percent after a prompt change. The raw token dashboard looked normal because volume was down; the cost-per-outcome metric caught it immediately.

Dashboards to Build

• Daily spend by model, feature, and tenant.
• Cost per outcome trended over 30 days.
• Cache hit rate by endpoint (any endpoint below 15 percent for high-volume queries is worth investigating).
• P95 input token length by endpoint (growing prompts are a smell).
• Top 10 users by token spend this week vs. last week (catches abuse and bugs).

Tools: LangSmith, Helicone, and Langfuse all capture this data. You can also build it yourself if you have a solid event pipeline. The important thing is that it exists and that someone looks at it weekly.

Prompt Hygiene: The Free Wins Most Teams Skip

Before adding any infrastructure, audit your prompts. In every production system I have reviewed, there are prompt tokens being spent on things that do not contribute to output quality. Common ones:

• Redundant context injection: including the full conversation history on every turn when only the last 2 to 3 turns are semantically relevant. Fix with a sliding window or a summary of older turns.
• Bloated system prompts: a 3,000-word system prompt written in sprint 1 that was never trimmed. Audit each instruction: is it producing a measurable change in output? Remove what isn't.
• Structured output overhead: asking the model to return verbose JSON when you only need two fields. If you need {'{'}'status': 'approved', 'reason': '...'{'}'}, ask for exactly that schema, not a 15-field object.
• Few-shot examples too long: long few-shot examples are expensive. Trim them. Use shorter, more representative examples. Three tight examples usually outperform six verbose ones at a fraction of the token cost.

Run a prompt token audit quarterly. The free wins compound.

Output Length Control

Output tokens cost the same as or more than input tokens on most models. Set max_tokens explicitly on every call. If your feature produces summaries, define what 'summary length' means in your system prompt ('respond in 2 to 3 sentences, never more'). Vague prompts produce verbose outputs that cost more and are often lower quality.

Frequently Asked Questions

How do I know which LLM model tier is right for each task?

Start by categorizing your calls into three buckets: classification and routing (cheap, use mini/haiku), retrieval-augmented generation and structured extraction (medium, use a mid-tier model), and open-ended reasoning, complex multi-step tasks, and creative work (expensive, use frontier). Then measure output quality on a labeled eval set for each bucket. Drop to the cheaper model only when quality difference is below your acceptable threshold. Never guess: build a small eval harness and let the data decide.

What is the ROI of semantic caching for LLM apps?

For most production systems with repeat query patterns, semantic caching reduces inference costs by 20 to 40 percent and cuts latency on cached hits by 80 to 90 percent. The setup cost is roughly two to three days of engineering time for a Redis vector index plus cache middleware. It pays for itself in the first billing cycle for any system at meaningful scale. The ROI is lower for systems with highly unique, creative, or personalized queries and higher for systems with FAQ-style, documentation-lookup, or classification workloads.

How do I prevent a viral moment from causing a runaway LLM bill?

Three layers: a per-user per-minute rate limiter enforced atomically in Redis before the API call, a per-tenant daily token budget that alerts at 80 percent and hard-stops at 110 percent, and a system-wide circuit breaker that pauses all new LLM requests if aggregate hourly spend exceeds 2x the expected rate. Your LLM API provider's soft or hard cap is not sufficient on its own because it operates at the account level and does not protect you from a single tenant or a single bugged client consuming everything.

Should I fine-tune a model to reduce costs?

Fine-tuning is often oversold as a cost solution. It can help in specific situations: when you need very consistent structured output, when few-shot prompting is expensive due to example length, or when a task is extremely narrow and well-defined. But fine-tuning adds significant maintenance overhead: every base model update requires re-evaluation and potentially re-training. Start with prompt optimization, model routing, and caching. Fine-tune only when those levers are exhausted and you have a stable, well-labeled dataset. The majority of the teams I work with do not need fine-tuning to control costs.

What is a reasonable cost-per-user-per-month target for an AI feature?

It depends heavily on the feature, but here is a rough benchmark from production systems: a conversational assistant with moderate usage should cost between $0.50 and $3.00 per active user per month with a well-architected system. If you are above $5.00 per user per month for a standard assistant feature, you almost certainly have an architecture problem, not a pricing problem. Benchmark against your revenue per user: if LLM cost exceeds 15 to 20 percent of the subscription price for an average user, the economics are unsustainable and you need to re-architect before scaling.

Is GPT-4o-mini or Claude Haiku good enough for production AI features?

For the right tasks, yes, absolutely. These models are remarkably capable for classification, intent detection, summarization of short to medium text, structured data extraction, and RAG-based Q&A over clean documents. Where they fall short: long-context reasoning over ambiguous multi-document inputs, complex code generation, and nuanced multi-step task planning. Run your specific task through an eval with labeled examples before committing. Most teams are surprised by how much a smaller model can handle once they have a well-crafted prompt.

Architecture Now Beats Optimization Later

Every week you run an AI system without semantic caching, model routing, and per-user limits is a week of unnecessary spend that you cannot recover. The good news is that these are not hard problems once you understand the system. They are design decisions, and design decisions are best made early, before the traffic arrives and before the invoice is already in your inbox.

If you are building an LLM application and want to get the architecture right before it goes to production, or if you are already in production and costs are climbing faster than value, that is exactly the work I do through my AI Architecture advisory service. I review your system, identify the cost drivers, and give you a concrete plan to fix them. No agency overhead, no junior consultants: you get me, with 16+ years of production systems experience and the scars to prove it. Reach out via the contact page or go straight to the service page.

Work with me on your AI architecture