Self-Hosting an LLM in 2026: vLLM vs SGLang vs Ollama, and How to Pick

The problem

At some point every team that ships an AI feature asks: "why are we paying per-token for a model we could run ourselves?" The answer usually shows up six weeks later, in production, as an incident. Someone stood up Ollama on a single GPU box, pointed the app at it, and it worked fine in the demo. Then real traffic hit — twenty concurrent users, each with a long system prompt — and latency went from 400ms to 14 seconds. Nothing crashed. The server just serialized requests one at a time because nobody read past "ollama run".

Self-hosting isn't hard because running a model is hard. It's hard because the serving layer has to solve three problems simultaneously — batching concurrent requests, managing GPU memory for the KV-cache, and (increasingly) guaranteeing structured output — and the tool you pick for a demo is rarely the tool you want for production.

This post is a decision guide for engineers choosing between the three serving stacks you'll actually encounter: Ollama, vLLM, and SGLang. (TensorRT-LLM is real but NVIDIA-specific and heavier to operate; skip it unless you're already deep in NVIDIA tooling and need the last 10-15% of throughput.)

Why "just run the model" doesn't scale

A single forward pass through a transformer is cheap. What's expensive is the KV-cache — the growing set of key/value tensors for every token in every active conversation. Naive serving allocates a fixed, oversized memory block per request "just in case" it grows long, which means most of your GPU's memory sits idle, reserved but unused, while requests queue up waiting for a slot.

The three serving engines differ mainly in how cleverly they manage that memory and batch requests around it:

• Ollama wraps llama.cpp and is optimized for single-user, single-request ergonomics — ollama run model and you're talking to it in ten seconds. It supports concurrency, but the batching and memory-management story is deliberately simple, which is exactly what makes it fast to adopt and wrong for concurrent production traffic.
• vLLM introduced PagedAttention, which manages the KV-cache like an OS manages virtual memory — in fixed-size pages that can be allocated non-contiguously. That eliminates the fragmentation that kills naive servers and is why vLLM became the default production choice.
• SGLang takes a different angle: RadixAttention. It stores the KV-cache in a radix tree so that any two requests sharing a prefix (the same system prompt, the same few-shot examples, the same conversation history) reuse the cached computation instead of recomputing it. For agents and RAG pipelines — where every call resends a long, mostly-identical prompt — this can beat vLLM on throughput because it's avoiding work vLLM still does.

Neither is strictly better. PagedAttention optimizes memory efficiency for arbitrary, unrelated requests. RadixAttention optimizes for the specific (very common) case where requests share structure.

Setting one up: vLLM

pip install vllm

# Serve an OpenAI-compatible endpoint
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192 \
  --max-num-seqs 64

Call it exactly like the OpenAI API:

from openai import OpenAI

client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")

resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Summarize this incident report..."}],
    max_tokens=500,
)
print(resp.choices[0].message.content)

The flags that actually matter in production:

• --gpu-memory-utilization: fraction of GPU memory vLLM is allowed to claim for the KV-cache pool. Too low and you under-batch; too high and you OOM when a burst of long requests lands. Start at 0.85-0.90 and watch for OOMs before pushing higher.
• --max-num-seqs: the batching ceiling. This is your actual concurrency limit — set it based on load testing, not guesswork.
• --max-model-len: caps context length per request. Set this to what your application actually sends, not the model's theoretical max — a smaller cap means more requests fit in the same KV-cache pool.

Setting one up: SGLang

pip install "sglang[all]"

python -m sglang.launch_server \
  --model-path meta-llama/Llama-3.1-8B-Instruct \
  --port 30000

SGLang also exposes an OpenAI-compatible endpoint, but its real value shows up when you use its native frontend for structured, multi-step generation:

import sglang as sgl

@sgl.function
def extract_ticket(s, ticket_text):
    s += sgl.user(f"Extract fields from this support ticket:\n{ticket_text}")
    s += sgl.assistant(
        "priority: " + sgl.gen("priority", choices=["low", "medium", "high"]) + "\n"
        "category: " + sgl.gen("category", choices=["billing", "bug", "feature_request"]) + "\n"
        "summary: " + sgl.gen("summary", max_tokens=50)
    )

sgl.set_default_backend(sgl.RuntimeEndpoint("http://localhost:30000"))
state = extract_ticket.run(ticket_text="Customer can't complete checkout...")
print(state["priority"], state["category"], state["summary"])

The choices=[...] constraint is enforced at the token level — the model literally cannot generate an out-of-set value. If your pipeline shares a fixed prefix (a classification prompt run against thousands of tickets, an agent replaying the same tool schema every turn), RadixAttention caches that shared prefix once and reuses it across every call.

Comparison table

	Ollama	vLLM	SGLang
Best for	Local dev, prototypes, single-user	Production, high-concurrency, arbitrary requests	Agents, RAG, shared-prefix / structured workloads
Core mechanism	llama.cpp wrapper, simple batching	PagedAttention (paged KV-cache)	RadixAttention (prefix-sharing KV-cache)
Setup time	Minutes	Tens of minutes (needs GPU + CUDA setup)	Similar to vLLM
Concurrency ceiling	Low by default, narrow tuning surface	High, tunable via --max-num-seqs	High, best when requests share structure
Structured output	Basic (via Modelfile/grammar)	Via plugins (Outlines, guidance)	Native, first-class (sgl.gen(choices=...))
GPU required	No (CPU-capable via llama.cpp)	Yes, for real throughput	Yes
Operational maturity	Very high (huge community)	Very high (default in most inference platforms)	High, growing fast, smaller ecosystem than vLLM

Checklist before you pick one

• Do you actually have concurrent users, or is this single-user internal tooling? If the latter, stop reading and use Ollama — anything else is over-engineering.
• Do your requests share a long, mostly-fixed prefix (system prompt, tool schema, few-shot examples)? If yes, benchmark SGLang against vLLM on your actual prompts before committing — the throughput gap is workload-dependent, not universal.
• Have you load-tested with your real prompt-length distribution, not a synthetic short prompt? KV-cache pressure comes from long contexts, and a benchmark with 50-token prompts will lie to you about production behavior with 4,000-token ones.
• Do you need hard-constrained structured output (enum fields, JSON schema)? SGLang's native constraint decoding is more reliable than bolting a grammar library onto vLLM after the fact.
• Have you set --max-model-len to your application's real max, not the model card's max? This single setting has the biggest effect on how many concurrent requests fit in memory.

Common pitfalls

Benchmarking with one request at a time. Every serving engine looks similar at concurrency=1. The entire reason PagedAttention and RadixAttention exist is to handle concurrency; test at the concurrency you'll actually run in production, or your numbers are meaningless.

Ignoring GPU memory headroom until an OOM in production. --gpu-memory-utilization 0.95 looks great in a load test that never varies prompt length. Then a user pastes a 6,000-token document and the KV-cache pool has nowhere to grow. Leave real headroom, and alert on GPU memory utilization, not just request latency.

Assuming Ollama's OpenAI-compat endpoint means production parity. The endpoint shape being compatible says nothing about the batching and memory management underneath it. Swapping base_url from vLLM to Ollama does not give you the same throughput characteristics — treat it as a dev/prod parity gap, not a drop-in replacement.

Skipping quantization tradeoffs. All three engines support quantized weights (AWQ, GPTQ, FP8), which changes both memory footprint and output quality. Decide your quality bar with real eval data before quantizing — don't discover a quality regression from users.

Not planning for model updates. Self-hosting means you now own the upgrade path when a better open model ships. Budget for periodic re-evaluation against your eval suite, not just a one-time "we picked Llama 3.1 and we're done."

Next steps

Self-hosting makes sense once your token volume and latency requirements justify the operational cost of owning a GPU fleet — not before. If you're still early, a hosted API is almost always cheaper in engineering time even if it's more expensive per token. When you do cross that line, start with vLLM for general workloads or SGLang if your traffic is agent- or RAG-shaped with heavy prompt reuse, and load-test both against your actual prompt distribution before betting infrastructure on either.

If you're building the muscle to make calls like this — not just following a tutorial, but reasoning about tradeoffs under real production constraints — that's exactly what we drill in the Joinloop AI engineering cohort.