How to Avoid LLM Vendor Lock-In With a Model Abstraction Layer

The Short Answer: Wrap the Model on Day One

You avoid LLM vendor lock-in by placing a thin, stable interface between your application logic and every model call, so the rest of your codebase never imports an SDK directly. That single architectural decision lets you swap providers, add fallbacks, and route different tasks to different models without touching business logic.

I am Mahmoud Zalt, an independent senior AI systems architect with 16 years of production engineering experience since 2010. I designed Apiato, an open-source framework built around swappable, decoupled components, and I apply the same discipline at Sista AI, the company I founded, where autonomous agents have run in production for the past year. Through my AI Architecture advisory work I have helped teams untangle exactly this problem after they discovered their entire pipeline was OpenAI-shaped. The fix is always the same: the abstraction should have gone in first. Here is how to do it right, and what to avoid.

Why Teams Get Locked In (It Is Not the API Key)

The surface-level problem looks like scattered openai.chat.completions.create() calls in 40 files. The deeper problem is that the shape of one provider bleeds into business logic: prompt templates written for GPT-4o tool-calling syntax, retry logic that assumes a specific HTTP error schema, cost tracking built around OpenAI token counting, and eval fixtures that hardcode model names.

I have seen three failure modes repeatedly:

• Direct SDK saturation. The vendor SDK is imported everywhere. Swapping means a project-wide find-and-replace followed by testing every call site.
• Prompt coupling. Prompts use provider-specific features (function-calling JSON schema for one vendor, tool_use blocks for another) with no abstraction over the difference.
• Implicit capability assumptions. Code assumes a context window size, a specific tokenizer, or a JSON-mode guarantee that does not exist on every provider. When you swap, silent regressions appear.

The fix is not a big-bang refactor. It is a boundary you draw once, early, and hold.

What the Abstraction Layer Should Look Like

The interface should be stable across providers and narrow enough that adding a new backend takes an afternoon. Here is the minimum surface I use on every engagement:

// TypeScript interface -- provider-agnostic
interface ModelClient {
  complete(req: CompletionRequest): Promise;
  stream(req: CompletionRequest): AsyncIterable;
  embed(req: EmbedRequest): Promise;
}

interface CompletionRequest {
  messages: Message[];
  tools?: Tool[];        // normalized, not provider-specific
  maxTokens?: number;
  temperature?: number;
  metadata?: Record; // task tag, user id, etc.
}

Key decisions in the design:

• Normalize tool/function definitions. Write them once in your schema; each adapter converts to the provider format (OpenAI tools array, Anthropic tool_use, Gemini functionDeclarations). This is the most important normalization to get right.
• Return normalized usage objects. Every response includes { inputTokens, outputTokens, cachedTokens, costUsd } computed inside the adapter, not in calling code.
• Pass metadata through. The metadata field on the request becomes the source of truth for tracing, cost allocation, and evals. Every adapter attaches it to the span it opens.
• Do not hide streaming. Expose stream() as a first-class method. Wrapping streaming poorly (buffering the whole response) defeats one of the main reasons to stream.

What the Abstraction Should NOT Hide

Over-abstraction is as dangerous as none. I have reviewed systems where the wrapper tried to normalize away every difference between models and ended up producing prompt regressions on half the providers.

Do not hide:

• System prompt position semantics. Anthropic requires the system prompt in a dedicated field; OpenAI treats it as a message with role system. The adapter handles this translation, but your prompt-building code should be explicit about which part is the system prompt rather than appending it as a regular message.
• Context window limits per model. Expose client.contextWindow so callers can truncate or summarize before they call, not after they hit a 400. Hiding this causes silent truncation on cheaper or older models.
• Native cache-control headers. Anthropic has explicit prompt caching with cache_control blocks. OpenAI caches automatically above a prefix threshold. If you normalize these away, you lose the ability to optimize cache hit rates deliberately. Expose a cacheable: true hint on the request and let the adapter map it correctly.
• Provider-specific failure modes. Do not swallow a context_length_exceeded error into a generic ModelError. Preserve the error class so callers can decide whether to retry with truncation versus retry with a larger model.

The rule: translate syntax, never erase semantics.

Routing Per Task and Falling Back on Failure

Once the abstraction exists, you can route intelligently. Here is the routing logic I implement on most production systems:

Fallback logic lives inside a ResilienceDecorator that wraps any ModelClient:

class ResilienceDecorator implements ModelClient {
  constructor(
    private primary: ModelClient,
    private fallback: ModelClient,
    private opts = { maxRetries: 2, fallbackOn: [429, 503] }
  ) {}

  async complete(req: CompletionRequest) {
    try {
      return await withRetry(() => this.primary.complete(req), this.opts.maxRetries);
    } catch (err) {
      if (this.opts.fallbackOn.includes(err.statusCode)) {
        return this.fallback.complete(req);
      }
      throw err;
    }
  }
}

Compose this at the application bootstrap level, not inside feature code. A feature should never know which provider answered its request.

Evals, Observability, and Making the Swap Safely

The abstraction layer makes swapping possible. Evals make it safe. These are the two things most teams skip, and skipping either turns 'swap in an afternoon' into 'redeploy and hope'.

What to instrument inside every adapter

Every adapter should open a trace span on entry and record: provider name, model ID, input/output token counts, cached token counts, latency to first token, latency to last token, error class if any, and the task tag from metadata. Use OpenTelemetry or a purpose-built LLM observability tool (Langfuse, Helicone, Braintrust). The task tag is what lets you filter cost dashboards to 'classification tasks' and compare GPT-4o-mini vs Haiku side-by-side before you make the switch permanent.

Eval structure before a provider swap

A minimal eval suite for a swap has three layers:

1. Unit evals (offline). 50 to 200 golden input-output pairs per task type. Run against both the incumbent and the candidate. Score with an LLM judge or regex, depending on task. Gate: candidate must match or beat incumbent on accuracy, and cost must improve or stay flat.
2. Shadow traffic. Route 5% of live requests to the candidate adapter, log both responses, do not serve the candidate. Let it run for 48 hours. Compare outputs programmatically.
3. Canary release. 10% live traffic to candidate for 24 hours. Watch error rate, p99 latency, and user-facing quality signals. Ramp to 100% only when those three are stable.

Teams that skip straight to 100% cutover always find the regression they missed in offline evals. The shadow step costs almost nothing and catches the edge cases golden datasets do not cover.

MCP, Tool Calling, and the Abstraction Boundary

Model Context Protocol (MCP) is becoming the standard way to expose tools to agents. It introduces a specific challenge: the tool invocation format differs between providers at the wire level, but the tool definitions themselves are yours and should be provider-agnostic.

The right boundary: your MCP server exposes tools in canonical JSON Schema. Your model adapter translates those schemas into the provider's native tool format before each call, then translates tool_call results back to canonical form before returning. The MCP server never changes when you swap providers.

What this prevents: I have seen teams build MCP servers that output Anthropic-formatted tool_use blocks directly in the prompt. When they tried to route the same agent to GPT-4o, the function-call parsing broke silently and the agent began hallucinating tool responses. The fix was a two-hour adapter change, not a server rewrite. But it would have been zero hours if the boundary had been drawn correctly the first time.

One practical rule: your tool definitions live in one registry. Each adapter pulls from that registry and formats on the way out. Never let a tool definition contain provider-specific syntax.

Cost Controls, Security, and Human-in-the-Loop

The abstraction layer is the right place to enforce cross-cutting policies that apply regardless of provider:

Cost guardrails

Set per-task budget limits at the router level, not in the application. For example: classification tasks are budgeted at 500 input tokens max; if a caller passes a longer input, the router truncates and logs a warning before the call goes out. This prevents a bug in a caller from generating a $400 bill overnight. Track running cost per metadata.userId and per metadata.taskType. Alert when any bucket exceeds its hourly budget by 3x.

Input and output validation

Every request through the abstraction should pass a prompt injection scan (a simple heuristic classifier or a dedicated model call) before reaching the primary model. Every response should be validated against the expected schema before being returned. These two checks are cheap and prevent the most common production incidents I have seen: injected instructions that change agent behavior, and malformed JSON that crashes downstream parsers.

Human-in-the-loop hooks

For any action that is irreversible (sending an email, writing to a database, making an external API call), the abstraction layer should support an explicit requiresApproval: true flag on the CompletionRequest. When set, the response is held in a pending queue and not executed until a human confirms. The queue is provider-agnostic because the check happens after the model responds, inside the abstraction, before execution.

Frequently Asked Questions

Should I use LangChain or LiteLLM instead of rolling my own abstraction?

LiteLLM is a reasonable starting point if your team is small and moving fast. It normalizes the API surface across 100+ providers and handles retries. The downside: it is a large dependency with its own abstractions layered on top of yours, and when it breaks or lags a provider update, you are blocked. I use LiteLLM for prototypes, then graduate to a thin in-house adapter for anything that will run in production at scale. LangChain adds too much opinion about your entire pipeline and is hard to surgically remove later.

Does this mean I have to evaluate every model for every task?

No. Start with one model everywhere. The point of the abstraction is that you can differentiate later when you have data. Run your first eval pass after 30 days of production traffic, when you have real distributions of inputs. Optimizing routing prematurely is wasted effort. Optimizing after you have usage data usually cuts costs by 40-60% on high-volume classification or extraction tasks.

How do I handle provider outages without the fallback adding too much latency?

The ResilienceDecorator should use a circuit breaker pattern, not just retry. After 5 consecutive failures from the primary in a 60-second window, open the circuit and route directly to the fallback for the next 5 minutes without attempting the primary. This eliminates the retry latency for users during an outage. Reset the circuit on a successful primary call after the cooldown. Libraries like cockatiel (Node) or resilience4j (JVM) implement this correctly.

What about fine-tuned models? Do they fit into this pattern?

Yes. A fine-tuned model is just another adapter implementation. Point it at the fine-tune endpoint, implement the same ModelClient interface, and register it in your router for the specific task type it was trained for. The rest of the system does not change. I often see teams treat fine-tuned models as special cases and wire them directly into feature code, which recreates the lock-in problem at a smaller scale.

Is it worth building this abstraction for a small internal tool with no real traffic?

If it will run in production for more than 6 months, yes. The abstraction takes half a day to write and an hour to test. The cost of adding it later, after 30 files are calling the SDK directly, is 3 to 5 days of careful refactoring plus a full regression test pass. I have done both. The upfront investment wins every time.

How do I handle provider-specific features like Anthropic computer use or OpenAI o1 extended thinking?

Expose them as optional capability flags on the request: capabilities?: { computerUse?: true, extendedThinking?: true }. The router checks whether the selected adapter supports the requested capability and either routes to a compatible provider or rejects early with a clear error. This keeps experimental features accessible without coupling your core pipeline to them. When a capability becomes standard, promote it to the base interface.

Ready to Build a Provider-Agnostic AI System?

The abstraction layer is not a big-bang project. It is a boundary you draw once, hold consistently, and build on. If you are starting a new AI product, the right time to add it is before you write your first production prompt. If you are already locked in, the right time is now, starting with the highest-traffic call site.

I work with engineering teams as an independent AI Architecture advisor, helping them design systems that stay flexible as models, providers, and costs shift. If your team is building something that needs to survive the next 12 months of LLM market change, I can help you get the architecture right early. Reach me at the contact page or review my background and open-source work first.

Work with me on your AI architecture