When One Router Becomes a Control Tower

We’re examining how Ollama’s HTTP route layer coordinates inference work. Ollama serves local and cloud-backed models through native APIs as well as OpenAI-compatible and Anthropic-compatible endpoints. Its server/routes.go file is the boundary where requests are validated, models are resolved, runners are scheduled, and results are streamed—a critical role for the project, but one concentrated in a 2,260-line file.

That concentration is not automatically a design failure. A router must connect policy, transport, and scarce execution resources. The risk is that those decisions become implicit inside large handlers, untyped messages, and unbounded fan-out. I’m Mahmoud Zalt, an AI solutions architect, and we’ll use this file to establish one practical lesson: integration code stays safer when it makes decisions, protocols, and resource budgets explicit. We’ll start with the strongest existing boundary, then follow the hidden decisions that make the route layer difficult to change.

Reserve Scarce Resources Last

server/routes.go behaves like a control tower. It applies CORS and host policy, dispatches generation, chat, embedding, model-lifecycle, compatibility, and cloud routes, then connects inference requests to the scheduler and llm.LlamaServer. The useful architectural question is not whether all of these concerns touch HTTP; they do. It is whether the file gives their consequential decisions clear boundaries.

HTTP request
   |
   v
Gin router: Server.GenerateRoutes()
   |-- native and compatibility endpoints
   |-- model and cloud routes
   v
scheduleRunnerForModel()
   v
Scheduler -> llm.LlamaServer
   v
streamed NDJSON or buffered JSON

scheduleRunnerForModel is one of the file’s strongest boundaries. It checks a model’s capabilities and consolidates options before requesting a runner from the scheduler:

if err := m.CheckCapabilities(caps...); err != nil {
    return nil, nil, nil, fmt.Errorf("%s %w", m.Name, err)
}

opts, err := s.modelOptionsWithEmbeddingBatchDefault(m, requestOpts, embeddingBatchDefault)
if err != nil {
    return nil, nil, nil, err
}

runnerCh, errCh := s.sched.getRunner(ctx, m, opts, keepAlive, numCtxAuto, numBatchAuto, shift)
select {
case runner := <-runnerCh:
    return runner.llama, m, &opts, nil
case err = <-errCh:
    return nil, nil, nil, err
}

This order protects a scarce resource. An incompatible model is rejected before it can consume scheduler or runner capacity. The helper also carries the resolved model through validation, option merging, and allocation rather than loading metadata again. That makes it a useful reservation boundary: validate deterministic requirements first; acquire shared resources second.

Turn Branches Into Named Decisions

The handlers must still decide which requests reach that reservation boundary. That is where the control tower becomes hard to operate. GenerateHandler is reported at 410 lines with cyclomatic complexity 48 and cognitive complexity 78; ChatHandler is 320 lines, 43, and 70; EmbedHandler is 240 lines, 32, and 49.

Cyclomatic complexity estimates independent decision paths. Cognitive complexity estimates how hard nested decisions are to follow. Neither metric is a verdict, but together they explain why a small change can require reconstructing local versus cloud routing, runner selection, load and unload behavior, capabilities, templates, parsers, raw mode, streaming, and error handling.

Repeated TopLogprobs checks in generation and chat illustrate the smallest worthwhile extraction. A shared validator is not about saving lines; it makes one API contract authoritative:

func validateTopLogprobs(topLogprobs int) error {
    if topLogprobs < 0 || topLogprobs > 20 {
        return errors.New("top_logprobs must be between 0 and 20")
    }
    return nil
}

The same principle scales beyond validation. Resolve the request once, then represent source selection, required capabilities, chat mode, parser choice, prompt strategy, streaming mode, and consolidated options as typed internal values. The goal is not to split long functions mechanically. It is to give each consequential decision a name and a focused test before execution begins.

Treat Streams as Protocols

Once execution starts, the route layer has another boundary to clarify: the messages crossing from background work to HTTP responses. Generation, chat, pull, and push use channels, but streamResponse receives chan any and interprets a gin.H map containing error and optional status as an error.

if h, ok := val.(gin.H); ok {
    if e, ok := h["error"].(string); ok {
        status, ok := h["status"].(int)
        if !ok {
            status = http.StatusInternalServerError
        }
        // Write JSON before streaming, NDJSON after it begins.
    }
}

The behavior recognizes a real HTTP constraint. Before response bytes are written, the handler can send a normal JSON error with an HTTP status. After streaming starts, that status is committed, so an error must be sent as another NDJSON event.

The weakness is the implicit contract. Producers and consumers must remember that ordinary values are response payloads while certain maps are failures. An unexpected value is only discovered at runtime. A typed event containing a response payload, an error, and an optional status would make that agreement compiler-visible; endpoint-specific event types could tighten it further.

The channel choice also affects capacity. An unbuffered channel prevents producers from running ahead of the HTTP consumer, but a slow client can stall callback progress. With many long-lived streams, socket usage, goroutine counts, and disconnect handling are part of inference reliability, not merely transport details. Typed events make the relevant cases—early error, late error, normal closure, and cancellation—straightforward to specify and test.

Give Concurrency a Budget

Explicit protocols reveal the next question: how much work can one request create? EmbedHandler safely stores embeddings in distinct slice indexes and updates token totals atomically. It also narrowly retries runner HTTP 413 responses only when truncation is allowed, avoiding blind retries for unrelated validation or media failures.

But the handler starts one errgroup goroutine per input item. A large request can therefore create many tokenization and embedding calls while retaining inputs, decoded media, and vectors. A small admission-control boundary prevents a single request from flooding the runner scheduler:

var g errgroup.Group
g.SetLimit(8)
embeddings := make([][]float32, len(input))

The value 8 is a proposal, not a universal setting; it must be benchmarked against the runner, hardware, and workload. The essential improvement is making the budget exist and making it observable. The same applies to truncateNativeChatMessages, which may repeatedly rebuild, render, and tokenize shorter suffixes of a long conversation, with reported worst-case work of approximately O(m² × render/tokenize cost).

Percentiles such as p95 expose slow outliers that averages hide, including cold model loads, long prompts, cloud calls, and slow stream consumers. Capacity should also be explicit at the HTTP boundary: this file does not visibly configure ReadHeaderTimeout, ReadTimeout, WriteTimeout, or IdleTimeout, and cloud calls use http.DefaultClient. An explicit server configuration, an explicit API multiplexer, and separately protected diagnostics would make the deployment boundary easier to audit.

A Safer Flight Plan

The primary lesson is that a router can coordinate many concerns safely only when it makes its decisions, protocols, and resource limits explicit. Ollama’s route layer already demonstrates the foundation: deterministic capability checks precede runner allocation, context flows into scheduling, retry behavior is constrained, and the runner boundary is clear. The remaining work is an incremental clarification, not a dramatic rewrite.

1. Resolve once. Bind and validate the request, normalize the runner, parse the model reference, and determine whether execution is local or cloud-backed.
2. Plan before reserving. Build typed internal values for capabilities, prompt and parser strategy, stream mode, and options; retain scheduleRunnerForModel as the resource-reservation boundary.
3. Budget and verify. Replace implicit stream messages with typed events, bound embedding fan-out, configure timeouts, and test stream timing and retry behavior at their boundaries.

Table-driven tests can lock down normalizeRunner aliases and the TopLogprobs bounds of 0 and 20. Handler tests can verify NDJSON framing and distinguish errors before bytes are written from errors emitted mid-stream. Embedding tests can confirm that only a 413 with truncation enabled retries once; concurrent cases under Go’s race detector can protect indexed writes and cancellation.

As this boundary evolves, the forward-looking implication is clear: new compatibility modes, cloud paths, and model features should become additions to explicit plans and protocols—not another set of branches every engineer must hold in their head.