The Bill You Actually Have

The first step is an honest accounting. Most teams have a number from finance that is a fraction of the real AI cost. Inference is the visible portion; the iceberg includes embeddings, fine-tuning, observability, vector storage, batch jobs, the staging environment that nobody throttled, evaluation runs, and the engineering time spent iterating on prompts.

Start every cost engagement by reconciling three views: the provider invoices (Anthropic, OpenAI, Bedrock, Azure OpenAI), the cloud bill (egress, GPU, vector DB), and the observability data (LangSmith, Langfuse, Helicone, Braintrust). The deltas between them are where the surprises live. I have never run an audit that did not find at least one orphaned API key driving five-figure monthly spend on a feature shipped 18 months ago and forgotten.

• Provider invoices broken out by model, environment, and API key
• Egress and GPU on the cloud bill, often 10-25% of the model bill itself
• Vector DB cost: Pinecone, Weaviate, Qdrant, pgvector storage and query
• Embedding spend: usually under-tracked because it runs in batch jobs
• Fine-tuning and base-model training: bursty, often hidden in research budgets
• Eval and red-team runs: especially expensive when they run on every commit
• Observability platform itself: LangSmith, Langfuse, Helicone bills
• Engineering time on prompt iteration, the largest hidden cost on small teams

Lever 1: Model Routing (Typical Savings 40-70%)

The single highest-leverage move is routing. Most teams use one model (usually GPT-4o, Claude Sonnet, or Claude Opus) for every request, including the 60-80% of traffic that a 5-10x cheaper model handles perfectly. A 2026 cost comparison: Claude Sonnet 4.5 input is roughly 5x the price of Haiku; GPT-4o is roughly 10x the price of GPT-4o-mini. Routing 70% of traffic to the small model and keeping the frontier model for the genuinely hard 30% typically cuts 50-65% off the bill while quality on the hard cases stays identical.

Routing is harder than it sounds because the router has to be cheap, fast, and correct. The published patterns that work: small classifier model (Haiku, Mini, Flash, or even a fine-tuned BERT) as the front door, prompt-difficulty heuristics (length, has-code, requires-multi-step), task-type routing (extraction vs reasoning vs generation), and cost-aware fallback (try cheap model first, escalate on low-confidence).

• Front-door classifier: Haiku, GPT-4o-mini, or Gemini Flash classifies the task type and difficulty
• Task-type routing: extraction and classification to small models, reasoning to mid-tier, novel composition to frontier
• Cost-aware fallback: try small model, escalate if confidence below threshold or output fails validation
• Per-user routing: free tier on cheap models, paid tier on premium, enterprise on dedicated
• Tools: Portkey, OpenRouter, Martian, LiteLLM, or LangChain RouterChain for orchestration
• Open-source local fallback: 7B-14B models on your own GPU for the 10% of traffic where API cost dominates
• Watch out: routing latency tax, classifier errors, and quality regression on edge cases

Lever 2: Prompt Caching (Typical Savings 45-90% on Cache Hits)

Prompt caching is the single biggest 2024-2025 LLM cost innovation. Anthropic prompt caching, OpenAI cached inputs, and Gemini context caching all let you mark a long stable prefix (system prompt, tool definitions, RAG context, conversation history) and pay a fraction of the input token cost on subsequent calls that reuse it. Anthropic charges 1.25x for the cache write and 0.1x for the cache read, meaning a cache hit on a 10,000-token system prompt is 90% cheaper than re-sending it. OpenAI gives 50% off cached input tokens.

Three deployment patterns dominate. First, system-prompt caching for any conversational product: the same persona and tool definitions get sent on every turn, so cache them. Second, RAG-prefix caching: retrieved documents that get reused across users within a TTL window. Third, conversation-history caching: each turn caches the previous turns, so a 20-turn dialogue pays full price once and 10% on every subsequent turn.

• System prompts and tool definitions: cache aggressively, they are the same on every request
• RAG document blocks: cache documents that appear in many retrievals, especially evergreen company docs
• Conversation history: each turn caches what came before, dramatic savings on long sessions
• Few-shot examples: cache the example set, not the per-request question
• TTL discipline: Anthropic 5-minute or 1-hour cache, OpenAI ~10-minute; design for that
• Cache key hygiene: tiny prompt edits invalidate the cache, so freeze stable prefixes
• Measure hit rate: aim for above 70% on cacheable workloads, alert when it drops

Lever 3: Batch APIs (Flat 50% Off Non-Real-Time)

Both OpenAI and Anthropic offer batch APIs that process requests asynchronously at a 50% discount, with results within 24 hours. For any non-interactive workload (offline classification, embedding generation, eval runs, document processing, summarization pipelines, content moderation backfills), this is a 50% cut for changing one API endpoint. It is the cheapest win in the entire playbook and the most under-used.

The migration is essentially mechanical. The hard part is identifying which workloads tolerate 24-hour latency, which usually turns out to be more than engineering thinks. Nightly ETL, weekly reporting, bulk embedding regeneration, eval pipelines, content moderation queues, and large back-office automations all qualify.

• Offline classification and tagging pipelines
• Embedding regeneration after index updates or model migrations
• Document summarization for archival and search
• Eval suite runs (nightly instead of per-commit)
• Bulk content moderation backfills
• Synthetic data generation for fine-tuning
• Migration is one API endpoint and a polling loop
• Watch out: batch results return within 24 hours, not always sooner; design SLAs accordingly

Lever 4: Prompt Compression (Typical Savings 20-50% on Input)

Most production prompts are 30-60% larger than they need to be. Iterating on prompts during development adds redundant instructions, dead few-shots, stale formatting rules, and verbose context blocks. A systematic prompt compression pass typically removes 20-50% of input tokens with no quality regression.

Tools and techniques: LLMLingua and LongLLMLingua for automated compression (paper-backed 2-20x compression ratios on long contexts with under 2% accuracy loss in many cases), human review of every prompt longer than 500 tokens, removing unused few-shots, replacing verbose instructions with terse versions, and structured output formats (JSON schema) instead of free-form prose.

• Audit every production prompt for redundant instructions and dead few-shots
• LLMLingua / LongLLMLingua for automated long-context compression
• Replace verbose "please be sure to" prose with terse imperatives
• Switch to structured output (JSON schema, tool calls) instead of natural language
• Cap conversation history with rolling summaries instead of full transcripts
• Move static instructions to system prompt where they get cached
• Use sentinel tokens and short variable names in templates

Lever 5: Semantic and Response Caching (20-40% on Repeatable Workloads)

Semantic caching stores past prompt-response pairs and returns the cached response when a new prompt is semantically similar to a prior one. For workloads with repetitive query patterns (customer support, FAQ chat, ecommerce recommendation, code documentation), 20-40% of traffic typically hits the cache with no model call at all.

Implementation tradeoffs are real. Hash-based exact-match caching is safe but rarely fires. Embedding-similarity caching fires more but risks returning a wrong answer to a subtly different question. Production teams use a hybrid: exact match first, embedding similarity with a high threshold second, and clear cache invalidation when underlying data changes.

• Exact-match response cache: deterministic prompts (canonical FAQ, system queries) should never re-run
• Embedding-similarity cache: GPTCache, Redis Vector, or Helicone Cache with a high threshold (0.95+)
• Tiered cache: exact match -> semantic match -> live call
• Invalidation discipline: any data change invalidates affected entries
• Per-user vs global cache: global for FAQs, per-user for personalized contexts
• Cache-write asymmetry: only cache responses you have validated, not raw model output
• Measure hit rate per route; below 10% means the cache is not paying for itself

Lever 6: Token Budget Discipline

Most production systems have no hard cap on output tokens, no max-context guard, and no per-call cost ceiling. The result is occasional thousand-dollar single requests where the model went into a loop, hallucinated a 50,000-token response, or recursed through too many tool calls. Token budgets are infrastructure, not optimization.

• max_tokens cap on every completion call, set conservatively per route
• Hard step limit on every agent loop (5-30 steps depending on task class)
• Per-request dollar cap enforced outside the model in the gateway layer
• Per-user and per-tenant rate limits and spend caps
• Streaming with early-termination when the response is structurally complete
• Truncate conversation history past N turns with summary preservation
• Reject prompts above a threshold size at the gateway with a clear error

Lever 7: Right-Size Embeddings and Vector Storage

Embedding spend grows quietly and stays hidden in batch jobs. Most teams over-embed. Three patterns drive 30-60% savings: smaller embedding models (text-embedding-3-small is 5x cheaper than large and 95% as good for most retrieval), dimension reduction (Matryoshka embeddings let you store 256 or 512 dimensions instead of 1536-3072 with minor recall loss), and chunking discipline (smaller index, fewer embeddings, faster queries).

Vector DB cost has its own dynamics. Pinecone p1.x1 pods at scale, Weaviate cloud, Qdrant cloud, and Azure AI Search all bill on dimension count, vector count, and query volume. Switching to pgvector on existing Postgres for sub-10M-vector workloads cuts the vector DB line item to roughly zero.

• Use text-embedding-3-small or open-source bge-small unless you have measured a quality difference
• Matryoshka truncation: 256-512 dimensions usually enough for production retrieval
• Chunking discipline: fewer, more meaningful chunks beat many small ones
• pgvector on existing Postgres for under-10M-vector workloads
• Re-embed only changed documents, not the full corpus on every model swap
• Index hygiene: drop unused namespaces, archive cold partitions, compact regularly
• Hybrid search (BM25 + vector): often beats pure vector at lower compute cost

Lever 8: Open-Weight Models for High-Volume Workloads

Above ~500M tokens per month, self-hosting open-weight models on dedicated GPU often becomes cheaper than hosted APIs. The 2026 open frontier (Llama 3.3 70B, Qwen 3 family, DeepSeek V3, Mistral) matches hosted models on most enterprise tasks. The crossover depends on workload: predictable steady-state high-volume favors self-hosting; bursty unpredictable low-volume favors APIs.

A common compromise: hybrid routing where high-volume predictable workloads (PII redaction, classification, summarization, retrieval reranking) run on local 7B-14B models, and the long tail of hard or low-volume calls bursts to a frontier API. vLLM and SGLang are the production-grade serving engines; Ollama is for prototypes.

• Crossover threshold: typically above 500M tokens/month for steady-state workloads
• vLLM or SGLang for serving, not Ollama or HuggingFace TGI in production
• Hybrid: small local model for high-volume simple tasks, frontier API for the long tail
• Quantization (INT8, INT4) cuts VRAM and cost, watch for quality degradation on reasoning
• Spot or reserved GPU on AWS, Lambda Labs, RunPod, or Modal for predictable workloads
• Track GPU utilization; below 40% means you are overprovisioned

Cost Governance and Org Discipline

Below a certain scale, cost optimization is a one-time audit. Above it, cost governance is a permanent discipline. The teams that stay cost-efficient have written budgets, real-time dashboards, and per-feature attribution. Without these, optimization gains erode in 6-12 months as new features ship without cost review.

• Per-feature or per-team budget allocation with monthly variance reports
• Cost-per-conversion, cost-per-resolved-ticket, cost-per-active-user dashboards
• Alerting on weekly cost growth above forecast (not on absolute thresholds)
• Quarterly model and provider mix review: rebalance as pricing changes
• Pre-launch cost review: every new AI feature scoped with projected per-call and monthly cost
• Cost annotation on every model swap, prompt change, infra migration
• Quarterly orphan-key cleanup: API keys for sunset features are the largest waste category
• Designate a cost owner: usually a senior engineer or fractional AI officer, with explicit authority

Common Mistakes That Cost Real Money

A short list of patterns I have audited out of production at multiple teams.

• Using Claude Opus or GPT-4o for tasks Haiku or Mini handle in 95% of cases
• Sending the entire conversation history every turn instead of summary-plus-recent-window
• Re-embedding the document corpus on every model upgrade instead of changed-document delta
• Running expensive eval suites on every commit instead of nightly or per-merge
• No max_tokens limit, allowing pathological 50,000-token completions
• Synchronous calls for jobs that could batch (50% savings left on the table)
• No prompt caching on system prompts that are identical across millions of calls
• Embeddings stored at 3072 dimensions when 512 would have worked
• Per-call API key per developer instead of pooled quotas, hiding total spend
• Local LLM deployment at sub-500M-tokens/month, where the GPU bill exceeds API cost

How I Engage on Cost Optimization

I run AI cost audits as fixed-scope engagements, typically 2-3 weeks. The deliverable is a written report ranking every lever by expected savings, the engineering effort to implement, the quality risk, and a 90-day execution plan with a finance-grade savings projection. Most audits identify 40-70% cuttable spend.

For ongoing programs, cost optimization is one workstream of a fractional AI officer engagement. The role owns monthly cost reporting, per-feature budget governance, and the discipline around new feature cost review. The first call is free and the right starting point is the most recent two months of provider invoices plus a list of the top three features by spend.

FAQ