Designing A Reliable LLM Gateway

Where the LLM Gateway Lives

In Langfuse, the LLM gateway is a shared server helper under a common package:

project-root/
  packages/
    shared/
      src/
        server/
          llm/
            fetchLLMCompletion.ts  <- shared server-side helper to fetch LLM completions

Conceptually, this module is a single counter at the post office. Every part of the system brings an envelope (the request), and the counter chooses the courier and route (provider, model, and parameters) before handing back a letter (the completion) in a consistent format.

The absence of the implementation forces us to focus on the one thing that matters most for long-term health: the boundary. The rest of this article is about designing that boundary so the gateway becomes a stable, evolvable abstraction instead of a brittle utility.

The Core Lesson: A Clean, Provider-Agnostic Boundary

The primary lesson is straightforward: treat your LLM gateway as a stable, provider-agnostic boundary with explicit types and error contracts. Once that boundary is right, you can change providers, add features, and improve reliability without touching the rest of your application.

One request and one response shape

A healthy gateway exposes a small set of types that everything else depends on. Provider specifics stay behind an interface.

// Illustrative design for fetchLLMCompletion.ts

export interface LLMCompletionRequest {
  model: string;
  prompt: string;
  maxTokens?: number;
  temperature?: number;
}

export interface LLMCompletionResponse {
  id: string;
  model: string;
  createdAt: Date;
  text: string;
}

export interface LLMCompletionProvider {
  complete(request: LLMCompletionRequest): Promise;
}

export async function fetchLLMCompletion(
  provider: LLMCompletionProvider,
  request: LLMCompletionRequest
): Promise {
  return provider.complete(request);
}

The rest of your system only knows about LLMCompletionRequest and LLMCompletionResponse. OpenAI vs Anthropic, HTTP vs SDK, streaming vs non-streaming—all of that is hidden behind LLMCompletionProvider. This is the key structural decision: a narrow interface that everything can rely on.

Centralization is not enough

It’s common to have a single helper that calls different providers, but still inlines all provider branches and error handling in one function. That feels convenient early on and becomes painful as soon as you grow.

By pushing variability (providers, configs, retries) behind the LLMCompletionProvider interface, you create a seam where you can swap implementations and add cross-cutting concerns without rewriting routes, jobs, or services.

Errors as part of the public contract

The other half of the boundary is failure. Provider SDKs throw different error shapes; callers should not have to understand them. Instead, normalize them into a small set of error kinds.

export type LLMErrorKind =
  | "rate_limit"
  | "timeout"
  | "provider_unavailable"
  | "invalid_request";

export class LLMCompletionError extends Error {
  constructor(
    public readonly kind: LLMErrorKind,
    message: string,
    public readonly cause?: unknown
  ) {
    super(message);
  }
}

export interface LLMCompletionProvider {
  complete(req: LLMCompletionRequest): Promise;
}

// Provider implementations translate their SDK errors to LLMCompletionError.

Once errors are standardized, callers can switch on kind and implement clear behavior: retry on "provider_unavailable", show a friendly message on "rate_limit", or surface input validation details for "invalid_request". That’s the difference between “sometimes it fails” and a predictable failure model.

Operations: Latency, Errors, and Observability

With the boundary in place, the next concern is how the gateway behaves in production. LLM calls are slow, expensive, and failure-prone compared to local logic, so the gateway must expose enough signals for you to operate it confidently.

Understand the real cost: network and provider latency

The main cost of fetchLLMCompletion is not CPU time; it’s network round-trips and provider compute time. You need metrics at the gateway to see what’s happening.

Three metrics are especially useful when emitted from the gateway layer:

• Total requests (for example, llm_completion_requests_total) to understand usage patterns, capacity, and cost.
• Latency (for example, llm_completion_latency_seconds) with percentiles, so you know what p95 and p99 look like for interactive flows.
• Error rate (for example, llm_completion_error_rate) to spot regressions, provider incidents, or misconfigurations.

Because every LLM call flows through the same gateway, this is the ideal place to record metrics and attach tracing spans such as llm.fetch_completion with attributes for provider and model.

Timeouts, retries, and resilience policy

Reliability features belong at the gateway boundary, not scattered across callers. Even without seeing the current code, we can shape the provider interface so it naturally supports:

• Timeouts so calls don’t hang indefinitely and user-facing paths can fail fast.
• Retries with backoff for transient issues like network glitches or provider 5xx responses.
• Clear idempotency behavior so repeated calls have predictable outcomes, even if the model is generative.

Logging and tracing without leaking sensitive data

LLM prompts often include personal or proprietary content, so naive logging is a liability. The gateway is again the right place to enforce safe patterns:

• Log metadata, not raw prompts: request id, provider, model, and coarse prompt size (for example, token count buckets).
• Emit structured error logs: error kind, provider, HTTP status, and whether the gateway will retry.
• Wrap each call in a trace span with attributes but avoid storing content in traces by default.

This keeps you operationally informed without turning logs and traces into a data-protection problem.

Testing the Gateway Contract

Once the boundary is explicit, testing becomes straightforward. You want to verify that the gateway’s contract is stable, independent of specific providers or network conditions.

Unit tests with mocked providers

The gateway should be testable with no real network calls. Because fetchLLMCompletion depends on an LLMCompletionProvider, you can inject a deterministic mock:

// Illustrative Jest-style unit test

it("returns a completion for a valid prompt", async () => {
  const provider: LLMCompletionProvider = {
    complete: async (req) => ({
      id: "test-id",
      model: req.model,
      createdAt: new Date("2024-01-01T00:00:00Z"),
      text: "Hello, world!",
    }),
  };

  const res = await fetchLLMCompletion(provider, {
    model: "gpt-4",
    prompt: "Say hello",
  });

  expect(res.text).toBe("Hello, world!");
});

This checks the gateway’s behavior and shapes without depending on a specific provider or environment.

Contract tests for failure modes

The same mocking approach lets you enforce your error contract:

• Have a mock provider throw a simulated rate-limit error and assert that the gateway surfaces an LLMCompletionError with kind = "rate_limit".
• Have a mock provider simulate slowness beyond your timeout and assert that callers see a timeout-specific error.

These tests guarantee that callers can rely on a stable set of error kinds, no matter how many providers or SDK versions sit underneath.

Edge cases worth encoding

Certain edge cases should be handled consistently and, where possible, covered by tests or at least clear design decisions:

• Empty or whitespace-only prompts.
• Prompts near model token limits.
• Invalid or deprecated model identifiers.
• Network failures, DNS issues, and provider-side rate limiting.
• Unexpected provider response schemas.

Each of these should map to a documented outcome: either a specific error kind or a validated constraint on inputs. That’s how a risky external dependency becomes a dependable internal API.

Conclusion: Making the Gateway a First-Class Component

Looking at the missing fetchLLMCompletion.ts in Langfuse leads to one clear conclusion: the function that talks to your LLM providers should be a carefully designed gateway, not a thin wrapper around an SDK. A clean, provider-agnostic boundary turns a fragile integration into a long-term asset.

To recap, the core lesson is that a stable gateway interface—explicit request/response types, normalized error kinds, and shared observability—is the foundation of a reliable LLM integration. Once that’s in place, you can evolve providers, policies, and performance without destabilizing the rest of your system.

Three practical takeaways you can apply in any codebase:

• Define the contract first. Introduce clear request, response, and error types at your LLM boundary, and hide provider quirks behind an interface like LLMCompletionProvider.
• Wire in operations at the gateway. Emit metrics for request volume, latency, and error rate, and wrap calls in trace spans so you can actually debug and tune the system.
• Test the gateway as a pure dependency. Make sure you can fully test success and failure paths with mocks and without hitting real providers; if you can’t, refactor toward a cleaner boundary.

If you treat your LLM gateway as a strategic integration point—not just a helper—you give every feature that depends on LLMs a stable foundation. That stability is what lets you add new providers, debug incidents faster, and meet reliability and compliance requirements as your product grows.