Skip to main content
Zalt Logo

Local LLM Deployment

Local LLM Deployment - Private, On-Prem, and Self-Hosted AI

Local LLM Deployment

Local LLM deployment has moved from curiosity to core infrastructure decision. When data cannot leave your perimeter, latency budgets are measured in milliseconds, or per-token economics at scale make hosted APIs untenable, running open-weight models on your own hardware becomes the default answer. The 2026 generation of open models (Llama 3.3, Qwen 3, DeepSeek V3, Mistral Large, Phi 4) now matches or beats frontier hosted models on most enterprise tasks.

The stack has matured in parallel. vLLM and SGLang push H100s past 16,000 tokens per second, llama.cpp runs 70B models on a Mac Studio, and Ollama wraps the whole thing in a one-line install. This page covers what local deployment actually means, how to pick models and hardware, which serving engine fits which workload, and where the real cost crossover lives.

What Local LLM Deployment Actually Means

A local deployment runs model weights on infrastructure you own or rent exclusively, with no inference traffic crossing a third-party API boundary. The driver is usually one of four constraints: data residency, regulatory exposure, latency, or unit cost at high token volume.

  • On-prem: weights and inference on hardware physically in your facility
  • Air-gapped: on-prem with no outbound network at all, used for classified and clinical-trial workloads
  • Private cloud: dedicated GPU instances on AWS, Azure, GCP, or sovereign providers (OVH, Scaleway)
  • Hybrid: small models local for PII redaction or routing, large frontier calls bursted to a hosted API
  • Edge: quantized 1B-8B models on laptops, phones, or industrial devices for offline use
  • Trigger threshold: above ~500M tokens/month, local typically beats hosted on raw cost
  • Compliance trigger: any workflow touching PHI, attorney-client data, or EU personal data at scale

Picking the Right Open Model

Model selection in 2026 is a four-axis problem: capability, size, license, and ecosystem support.

  • Llama 3.3 70B: strongest general-purpose open model, Llama Community License (free under 700M MAU)
  • Llama 3.2 1B/3B: edge and mobile, great for routing and classification
  • Qwen 3 (0.6B to 235B MoE): Apache 2.0, top multilingual, strong on code and math
  • DeepSeek V3 and R1 (671B MoE, 37B active): MIT-style, reasoning-class at fraction of GPT cost
  • Mistral Small 24B / Large 123B: Apache 2.0 for Small, commercial for Large, strong European-language
  • Mixtral 8x7B / 8x22B: mature MoE, easier to serve than dense 70B at similar quality
  • Phi 4 (14B): Microsoft MIT, punches above its weight for STEM at low VRAM
  • Rule: 7B-14B for summarization/extraction, 70B for reasoning, 200B+ MoE for frontier tasks

Hardware Sizing and VRAM Budgets

VRAM is the binding constraint. Formula: parameters × bytes-per-param + 20-30% for KV cache. FP16 = 2 bytes, INT8 = 1, INT4 = 0.5.

  • 7B FP16: ~16 GB VRAM, fits RTX 4090 (24 GB) or single A10
  • 13B FP16: ~28 GB, needs A100 40GB or 2x 4090 with tensor parallel
  • 70B FP16: ~140 GB, requires 2x H100 80GB or 4x A100 40GB with NVLink
  • 70B Q4_K_M: ~40 GB, fits single A100 80GB, H100, or RTX 6000 Ada (48 GB)
  • 405B FP16: ~810 GB, needs 8x H100 node or H200 cluster
  • RTX 4090 (~$1,600): best price-per-token for sub-13B serving and dev
  • A100 80GB ($1.50-$2/hr spot): cheapest per token for 70B quantized
  • H100 80GB ($2.50-$4/hr): 1.8-2.2x A100, mandatory for FP8 and high-throughput 70B+
  • M3 Ultra Mac Studio (192 GB unified): runs 70B Q4 at 8-12 tok/sec, silent, no rack

Quantization: GGUF vs AWQ vs GPTQ vs FP8

Modern 4-bit methods lose only 1-3% perplexity on most benchmarks. Tradeoff is nearly always worth it.

  • GGUF: llama.cpp/Ollama native, CPU+GPU hybrid, 1.5-bit through 8-bit, Q4_K_M is the sweet spot
  • AWQ: GPU-only, 4-bit, best accuracy retention for vLLM/TGI
  • GPTQ: older 4-bit GPU format, slightly lower quality than AWQ
  • FP8 (E4M3, E5M2): native on H100, near-FP16 quality at half memory, ~2x throughput
  • NVFP4/MXFP4: 4-bit floating point on Blackwell, new standard for 70B+ serving
  • Q8_0: barely distinguishable from FP16, use when you have VRAM headroom
  • Avoid Q2/Q3 for production: noticeable degradation on multi-step reasoning and code

Choosing a Serving Engine

  • vLLM: production default, PagedAttention, continuous batching, 15,000+ tok/sec on 7B
  • SGLang: ~29% faster than vLLM on smaller models, RadixAttention shines on shared prefixes
  • llama.cpp: minimal deps, GGUF native, only choice for Apple Silicon and CPU-only
  • Ollama: llama.cpp wrapped for dev ergonomics, prototyping only, not high-concurrency
  • TGI: now in maintenance mode, migrate to vLLM or SGLang
  • TensorRT-LLM: NVIDIA compiled engine, fastest on H100/Blackwell, painful to build
  • Rule: Ollama for laptops, vLLM for production, SGLang for agent/RAG with prefix reuse, llama.cpp for Macs and edge

Deployment Patterns by Industry

  • Healthcare (HIPAA): 7B-13B for chart summarization, 70B for differential diagnosis, BAA required for cloud
  • Legal: 70B class for contract review and discovery, air-gapped to preserve attorney-client privilege
  • Defense/intelligence: air-gapped classified networks, Llama 3 and Mistral preferred for license clarity
  • EU data sovereignty: on-prem or EU-headquartered providers only (US CLOUD Act exposes US-owned infra)
  • Financial services: hybrid, sensitive workflows local, low-risk drafting bursted to hosted
  • Pharma: clinical-trial data on-prem with strict retention and right-to-erasure
  • Public sector: sovereign cloud (OVH, Scaleway, IONOS) or on-prem; AI Act fines reach €35M or 7% global turnover

Cost Economics: Local vs Hosted

Hosted APIs win below ~100M tokens/month; local wins above 500M; the middle is hybrid.

  • Llama 70B FP16 on H100: ~118 tok/sec, ~$3/hr, ~$7 per million tokens
  • Llama 70B Q4_K_M on RTX 4090 spot ($0.40/hr): ~$2.65 per million tokens
  • A100 80GB spot $1.50-$2/hr: cheapest per-token for quantized 70B serving
  • Owned H100 node (8x H100, ~$300K capex): breaks even vs hosted GPT-class at ~2B tokens/month over 3 years
  • Mac Studio M3 Ultra ($7,000 one-time): runs 70B Q4 at single-user latency forever, zero per-token cost
  • Hidden costs: electricity (1-2 kW per H100), cooling, DevOps headcount, eval pipeline
  • Hosted advantage: zero capex, instant capacity, frontier-tier quality day one of a new release

Common Pitfalls

  • Underestimating KV cache: long contexts eat more VRAM than weights at high concurrency
  • Picking Ollama for multi-tenant production: does not batch like vLLM, falls over above a few users
  • Ignoring license fine print: Llama has 700M MAU clause, Mistral Large is non-commercial without paid agreement
  • Skipping eval harness: open models drift across quant levels, re-benchmark after every change
  • Over-quantizing reasoning models: DeepSeek R1, Qwen 3 reasoning lose noticeable quality below Q4
  • Assuming on-prem = compliant: HIPAA, GDPR, AI Act still require logging, access controls, DPIAs
  • Buying H100s for an RTX 6000 Ada workload: most common six-figure mistake of 2025-2026
  • Forgetting observability: prompt logs, token accounting, drift detection not optional at scale

FAQ

Can I run a 70B model on a single consumer GPU?

Yes with 4-bit quantization. Llama 3.3 70B at Q4_K_M fits ~40 GB, so a 48 GB RTX 6000 Ada or A6000 handles it. A 24 GB RTX 4090 cannot fit in VRAM alone and requires CPU offloading with speed penalties.

Is self-hosting actually cheaper than OpenAI/Anthropic?

Only above ~500M tokens/month sustained. Below that, hosted APIs win on TCO once you factor in DevOps, electricity, eval pipeline. Crossover comes earlier when latency or data residency is non-negotiable.

Which serving engine should I start with?

Ollama for prototyping. vLLM for multi-tenant or production SLAs. SGLang for agent-heavy workloads with shared prompt prefixes. llama.cpp for Apple Silicon or CPU-only.

Does on-prem make me HIPAA/GDPR compliant?

No. On-prem removes third-party data transfer risk, but you still need access controls, audit logging, prompt redaction, retention policies, governance. HIPAA still requires same safeguards regardless of where the model runs.

What is the quality gap vs GPT-4 class hosted?

For most enterprise tasks (summarization, extraction, classification, RAG, code completion) the gap is small or zero in 2026. Llama 3.3 70B, Qwen 3 72B, DeepSeek V3 trade blows with frontier hosted. Gap widens on novel reasoning and very long-context tasks.

GGUF or AWQ for production serving?

AWQ on vLLM/TGI with dedicated GPUs for max throughput with minimal accuracy loss. GGUF on llama.cpp/Ollama or CPU+GPU hybrid. AWQ edges out GGUF on pure GPU throughput.

Can a Mac realistically serve an LLM?

For single-user/small-team yes. M3 Ultra Mac Studio with 192 GB unified memory runs Llama 3.3 70B at Q4 around 8-12 tok/sec. Silent, under 300W, costs roughly one H100 hour per month over three years. Not for high-concurrency production.

Next step

Your situation isn't generic. Neither should the conversation be.

A short call to map what local llm deployment looks like for your team. No obligation, no pitch, just clarity.

Deploy LLMs on my hardwareSee Agent Development

Senior architect · 16+ years shipping · Direct, no agency layers