We’re analyzing how Ollama cleanly separates “the model” from “the rest of the system” through a single interface in llm/server.go. Ollama is a local LLM runtime that runs models on your own hardware. This file is the control tower: it defines the LlamaServer interface and the request/response types every API handler, CLI, or scheduler talks to. I’m Mahmoud Zalt, an AI solutions architect, and we’ll see how this small boundary quietly orchestrates complex LLM serving—and what patterns we can reuse when building our own systems.

One boundary, many concerns

llm/server.go sits between the API layer and the low-level llama runner. The ASCII map from the report shows its neighbors:

Project root (github.com/ollama/ollama)
|
+-- llm/
|   |
|   +-- server.go        (LlamaServer interface, DTOs, model loader)
|   +-- runner_*.go      (concrete NewLlamaServerRunner implementations)
|
+-- api/
|   +-- types.go         (api.Message, api.Options, api.Tools, api.ThinkValue)
|
+-- fs/
|   +-- ggml/
|       +-- ggml.go      (*ggml.GGML, Decode, KV metadata)
|
+-- ml/
|   +-- system.go        (SystemInfo, DeviceInfo, DeviceID)
|
+-- envconfig/
    +-- config.go        (KvCacheType and related configuration)

This file does not sample tokens or launch CUDA kernels. It defines the LlamaServer interface that every caller depends on:

type LlamaServer interface {
	ModelPath() string
	Load(ctx context.Context, systemInfo ml.SystemInfo, gpus []ml.DeviceInfo, requireFull bool) ([]ml.DeviceID, error)
	Ping(ctx context.Context) error
	WaitUntilRunning(ctx context.Context) error
	Completion(ctx context.Context, req CompletionRequest, fn func(CompletionResponse)) error
	Chat(ctx context.Context, req ChatRequest, fn func(ChatResponse)) error
	ApplyChatTemplate(ctx context.Context, req ChatRequest) (string, error)
	Embedding(ctx context.Context, input string) ([]float32, int, error)
	Tokenize(ctx context.Context, content string) ([]int, error)
	Detokenize(ctx context.Context, tokens []int) (string, error)
	Close() error
	MemorySize() (total, vram uint64)
	VRAMByGPU(id ml.DeviceID) uint64
	Pid() int
	GetPort() int
	GetDeviceInfos(ctx context.Context) []ml.DeviceInfo
	HasExited() bool
	ContextLength() int
}

Everything above this layer just knows:

• How to ask for completions, chat, embeddings, and tokenization.
• How to load models onto GPUs and wait for readiness.
• How to inspect memory, devices, and process state.

It never sees subprocess management, kv-cache types, or GPU allocation details. That is the primary lesson of this file: a thin, well-designed interface can tame the complexity of LLM serving by forcing all that chaos behind a single boundary.

Contracts, not runners

Once you see llm/server.go as a boundary, its design choices are easier to understand: it defines contracts—interfaces and DTOs—and delegates the rest.

From file path to running server

The journey from a model file on disk to a running LLM server goes through two small helpers.

1. Load the model file with minimal brains

func LoadModel(model string, maxArraySize int) (*ggml.GGML, error) {
	if _, err := os.Stat(model); err != nil {
		return nil, err
	}

	f, err := os.Open(model)
	if err != nil {
		return nil, err
	}
	defer f.Close()

	return ggml.Decode(f, maxArraySize)
}

LoadModel is intentionally dull: basic file checks, then hand off to ggml.Decode. All format complexity lives in fs/ggml. The boundary stays readable and easy to test.

2. Construct a runner with policy, then delegate

func NewLlamaServer(
	systemInfo ml.SystemInfo,
	gpus []ml.DeviceInfo,
	modelPath string,
	f *ggml.GGML,
	adapters, projectors []string,
	opts api.Options,
	numParallel int,
	config LlamaServerConfig,
) (LlamaServer, error) {
	slog.Info("using llama-server for model", "model", modelPath)

	trainCtx := f.KV().ContextLength()
	if opts.NumCtx > int(trainCtx) && trainCtx > 0 {
		slog.Warn("requested context size too large for model", "num_ctx", opts.NumCtx, "n_ctx_train", trainCtx)
		opts.NumCtx = int(trainCtx)
	}

	kvct := strings.ToLower(envconfig.KvCacheType())
	return NewLlamaServerRunner(gpus, modelPath, f, adapters, projectors, opts, numParallel, kvct, config)
}

The constructor does just enough before delegating:

• Log which model is being used (essential for debugging and audits).
• Clamp NumCtx to the model’s training context length, with a warning.
• Choose the kv-cache type from configuration.

Then it calls NewLlamaServerRunner, implemented in runner_*.go, where subprocesses and GPU details live. Policy stays at the boundary; mechanism is pushed into specialized components.

Guardrails in the types

The rest of the file is mostly types and helpers, but they encode important invariants: context limits, multimodal inputs, and metrics/logprobs. This is where the “simple interface” actually prevents real-world problems.

Context length as an invariant, not a convention

Large context windows drive memory and latency. The boundary makes one rule explicit:

• Effective NumCtx must not exceed the model’s training context length, if known.

NewLlamaServer enforces this by reading f.KV().ContextLength() from GGML metadata and clamping user-supplied options. That is a tiny amount of code that eliminates a whole class of “why is this model so slow / unstable?” tickets.

Multimodal messages without type soup

As soon as you accept images or audio alongside text, message structures tend to rot. Here, the boundary wraps the external api.Message in an internal Message plus MediaData:

type MediaKind string

const (
	MediaKindUnknown MediaKind = ""
	MediaKindImage   MediaKind = "image"
	MediaKindAudio   MediaKind = "audio"
)

type MediaData struct {
	Data []byte `json:"data"`
	ID   int    `json:"id"`
	Kind MediaKind
}

type Message struct {
	Role       string
	Content    string
	Thinking   string
	Media      []MediaData
	ToolCalls  []api.ToolCall
	ToolName   string
	ToolCallID string
}

func MessageFromAPI(msg api.Message) Message {
	media := make([]MediaData, len(msg.Images))
	for i, data := range msg.Images {
		media[i] = NewMediaData(i, data)
	}

	return Message{
		Role:       msg.Role,
		Content:    msg.Content,
		Thinking:   msg.Thinking,
		Media:      media,
		ToolCalls:  msg.ToolCalls,
		ToolName:   msg.ToolName,
		ToolCallID: msg.ToolCallID,
	}
}

MessageFromAPI is a customs officer: it inspects api.Message and repackages it into what runners expect, including assigning IDs to media bytes.

The report notes one gap: MediaKind is not automatically set based on the source. Downstream code could misclassify media. The fix fits the same philosophy:

• If downstream code must know something (image vs audio), encode it in the type.
• Do that at the boundary, once, inside conversion helpers like MessageFromAPI.

Metrics and logprobs as part of the contract

The interface doesn’t just stream text; completions and chat responses carry rich metadata. The completion side looks like this:

type TokenLogprob struct {
	Token   string  `json:"token"`
	Logprob float64 `json:"logprob"`
}

type Logprob struct {
	TokenLogprob
	TopLogprobs []TokenLogprob `json:"top_logprobs,omitempty"`
}

type CompletionResponse struct {
	Content               string        `json:"content"`
	DoneReason            DoneReason    `json:"done_reason"`
	Done                  bool          `json:"done"`
	PromptEvalCount       int           `json:"prompt_eval_count"`
	PromptEvalCachedCount *int          `json:"prompt_eval_cached_count,omitempty"`
	PromptEvalDuration    time.Duration `json:"prompt_eval_duration"`
	EvalCount             int           `json:"eval_count"`
	EvalDuration          time.Duration `json:"eval_duration"`
	Logprobs              []Logprob     `json:"logprobs,omitempty"`
}

Embedded here are three design decisions:

• Termination semantics via Done and DoneReason, not just “stream ended.”
• Performance metrics (counts and durations) on every streamed chunk.
• Optional introspection via Logprobs and TopLogprobs, only when requested.

Because these are part of the public contract, any consumer can turn them into counters and histograms later without touching runner internals. The boundary types encode what the team cares about operationally: how long things take, how many tokens, and why streams end.

Making the boundary safe to operate

Production LLM systems depend on good logging, but logging can easily leak secrets. This file wraps that concern in a small type, filteredEnv, that knows how to log itself safely:

type filteredEnv []string

func (e filteredEnv) LogValue() slog.Value {
	var attrs []slog.Attr
	for _, env := range e {
		if key, value, ok := strings.Cut(env, "="); ok {
			if filteredEnvLogKey(key) {
				attrs = append(attrs, slog.String(key, filteredEnvLogValue(key, value)))
			}
		}
	}
	return slog.GroupValue(attrs...)
}

func filteredEnvLogKey(key string) bool {
	return strings.HasPrefix(key, "CUDA_") ||
		strings.HasPrefix(key, "ROCR_") ||
		strings.HasPrefix(key, "ROCM_") ||
		strings.HasPrefix(key, "HIP_") ||
		strings.HasPrefix(key, "HSA_") ||
		strings.HasPrefix(key, "GGML_") ||
		slices.Contains([]string{
			"PATH",
			"LD_LIBRARY_PATH",
			"DYLD_LIBRARY_PATH",
		}, key)
}

func filteredEnvLogValue(key, value string) string {
	for _, token := range []string{"API", "KEY", "TOKEN", "SECRET", "PASSWORD", "PASS", "CREDENTIAL", "AUTH"} {
		if strings.Contains(strings.ToUpper(key), token) {
			return "[redacted]"
		}
	}
	return value
}

The protection is two-layered:

• Key filter: only environment variables related to GPUs/runtime or a small path whitelist are even considered for logging.
• Redaction: if a key name looks like it could contain secrets, its value is replaced with [redacted].

The rest of the codebase can then do:

slog.Info("llm env", "env", filteredEnv(os.Environ()))

and get useful information about CUDA or ROCm configuration without manually sprinkling redaction logic across every log line.

Scaling and explicit failure modes

As load and model sizes grow, a clean boundary either amplifies the pain or cushions it. llm/server.go quietly bakes in several scale-friendly choices.

Streaming and buffer bounds

LlamaServer uses callbacks for generation:

• Completion(ctx, req, func(CompletionResponse))
• Chat(ctx, req, func(ChatResponse))

This makes streaming the default, not an afterthought. The file also defines bounds for response buffering:

const (
	llamaServerStreamInitialBufferSize = 64 * 1024
	llamaServerStreamMaxBufferSize     = 8 * format.MegaByte
)

The details of the streaming loop live in the runner, but the boundary already encodes that responses are chunked, and a single line from the runner is capped. That keeps memory and backpressure in view at the contract level.

Metrics-first thinking

Because CompletionResponse and its chat sibling surface eval counts, durations, and termination reasons, the performance report can map them directly to metrics such as:

The contract exposes exactly what a platform team needs to reason about performance and capacity without instrumenting deep model code.

Named failure modes instead of “something broke”

Finally, the boundary refuses to treat all failures as the same. Two examples from the report:

• ErrLoadRequiredFull: signals that the model could not be partially loaded on GPU and the caller may need a different strategy.
• DoneReason enum: explains why a stream ended (stopped by token, length limit, connection closed, etc.).

That lets higher layers make concrete decisions: retry, downgrade model, show a user-friendly “context limit hit” message, or simply log and abort. A boolean Done plus an opaque error string would be much harder to act on.

What to steal for your own LLM interface

The value of llm/server.go is not in clever algorithms; it is in how ruthlessly it pushes complexity behind a small, opinionated interface. When you design the boundary between “LLM engine” and “application” in your own stack, there are concrete patterns worth copying.

1. Expose a single cohesive interface as your LLM boundary.

   Bundle loading, chat, completion, embeddings, tokenization, health, and memory inspection behind one interface similar to LlamaServer. Everything else—HTTP handlers, CLIs, schedulers—should depend only on that interface, never on concrete runners or GPU logic.

2. Enforce invariants at construction time.

   Clamp dangerous options like context size inside your constructor, using model metadata where possible. Catch misconfiguration once, at the boundary, instead of sprinkling checks across handlers.

3. Use DTOs that reflect operational reality.

   Design response types that carry termination reasons, token counts, and timing information. Include optional introspection (logprobs, top alternatives) but keep it explicit. This makes it trivial to wire metrics later.

4. Centralize safety and privacy behavior.

   Wrap sensitive data (like environment variables) in dedicated types that implement safe logging. Push key filtering and redaction into those types so call sites stay simple and consistent.

5. Give failure modes names.

   Define enums and sentinel errors for the kinds of failures and terminations you expect. Make sure callers can distinguish “ran out of context” from “connection dropped” from “GPU configuration incompatible.”

The primary lesson from this file is that a small, clear interface is the lever that moves the rest of the system. GPUs, runners, and formats will keep changing. A well-thought-out boundary like LlamaServer lets you absorb those changes without rewriting your entire stack.

The next time you work at the edge between your LLM engine and your application, ask yourself: does this boundary feel like a tidy power strip or a nest of wires? You can usually shift it toward the former just by tightening the interface and letting everything messy live behind it.