I use Ollama vs vLLM differently — one user or a crowd

Equal token speeds can hide different serving designs. See which engine fits a private Mac, an RTX 5090…

The short version

Ollama suits private, low-concurrency use; vLLM is designed for sustained serving when requests overlap.
A public RTX A6000 benchmark reached about 53 tokens per second with both engines under different quantization.
Choose by callers, usable memory, model behavior, and termination reliability—not headline throughput figures alone.

Your local AI server looks brilliant until everyone hits Send. Then Ollama vs vLLM becomes crowd control, not a benchmark debate: I give Ollama one person and vLLM the crowded room.

A public video benchmark shows the trap. With Qwen3-VL-4B-Instruct on an RTX A6000, vLLM FP8 and Ollama Q4 both generated about 53 tokens per second. Same headline speed, different configurations.

I still chase the biggest number before checking what produced it. But nobody has published a reproducible, peer-reviewed Ollama vs vLLM comparison holding model weights, quantization, hardware, context length, request mix, concurrency and quality constant. Every universal verdict wears a fake moustache.

Queues expose the difference

I use Ollama for private desktop work and vLLM for a shared GPU endpoint. Ollama makes GGUF models and Apple Silicon painless. vLLM’s complexity pays off when people or agents generate simultaneously. Once requests overlap, cache capacity and per-step memory determine how much live work the GPU carries. A faster card helps both engines without equalizing their serving designs. Count callers before CUDA cores.

According to the project’s release notes, Model Runner V2 is now vLLM’s default. It profiles CUDA graph memory first, then automatically sizes the KV cache from what remains. Estimating cache before knowing graph memory can make it collapse when serving starts. During generation, batch-sharded sampling divides logits work across tensor-parallel ranks. Each rank holds less logits memory per step, leaving room for more active sequences to share a decoding step. Think busy espresso bar, except everyone ordered one token and refuses to leave.

BOOST extends this memory strategy on Grace Hopper systems. The research integration places static-weight pages according to kernel access patterns, while its free KV-page pool understands GPU threadblock waves. One wave can access host memory as another works from HBM, keeping both paths busy. HBM-only serving restricts the working set; crude prefetching can waste host-memory bandwidth at the wrong time. We still don’t know whether BOOST’s gains transfer across GPU vendors, model architectures and graph-capture setups.

In that public benchmark, Ollama Q4 peaked near 14 GB of GPU memory while vLLM FP8 used roughly 38 GB with the same model, hardware and video workload. Because precision and quantization differed, the result mixes engine behavior with model representation. It demonstrates a memory trade-off, not a winner.

That matters when someone asks which engine belongs on an RTX 5090. No controlled test on that card holds the full stack constant. I still pick vLLM for sustained concurrent GPU serving because its cache sizing and sampling design target crowded workloads. For one person chatting at home, Ollama stays until the queue gets annoying.

Memory decides which local model survives

The best local LLM is the capable model that remains fully resident with room for context and runtime. “Usable memory” has ruined enough afternoons to need a warning label. ComfyUI, a desktop compositor or another process may already own the required chunk while your dashboard cheerfully claims memory is available.

On August 25, 2026, I tested both gpt-oss models on my M3 Max with 128 GB of unified memory. Both MXFP4 models remained fully resident.

The smaller gpt-oss:20b has about 21 billion parameters and generated around 74 tokens per second. The nearly 117-billion-parameter gpt-oss:120b still managed roughly 51 tokens per second—a modest slowdown for such a huge size increase.

Prompt processing hurt more. The smaller model processed around 756 tokens per second versus roughly 215 for the larger model. On the same setup, time to first token rose from about 3.7 seconds to 5.8 seconds. These results answer one narrow question about these models on my Mac, not which LLM runs best locally on every machine.

My RTX 5060 Ti provided comic relief. It has 16 GB of VRAM, but ComfyUI occupied almost all of it, leaving Ollama around 150 MB. The 20-billion-parameter model ran entirely on the CPU. Technically, the GPU was present. Spiritually, it had gone out for cigarettes.

Ollama’s MLX runner is improving the unglamorous memory work that keeps long Apple-hardware generations alive. During generation, its allocator collects cached free buffers for reuse. If that pool grows forever, it gradually consumes memory after intermediate allocations finish. Speculative decoding complicates cleanup because one round may emit several tokens. A modulo check waiting to hit an exact cleanup interval can skip past it. Ollama now detects when generation crosses the interval and releases cached buffers. Related changes shorten MLX array lifetimes, retain completed arrays within a scope and clear intermediate buffers between quantization items.

The project’s release notes also say MLX safetensors support in ollama create has graduated from experimental status. In the same release line, a cold /api/tags request on a large model library fell from around three seconds to about 0.3. The model stayed equally intelligent, but the menu opened roughly ten times faster. As an Italian, I respect prompt menu service.

Fast garbage is still garbage

Throughput means little when a response loops, loses its end marker or degrades under load. We lack a solid comparison of Ollama and vLLM output fidelity or failure rates during sustained concurrent production traffic. Issue trackers offer clues, not a universal verdict.

One Ollama issue reported concurrent Gemma requests losing their end-of-sequence marker and continuing toward a large token cap. Using the same GB10 machine and Ollama harness, the reporter’s concurrent Qwen trials completed cleanly. Solo Gemma requests also behaved normally. That suggests a model-specific interaction, not a general Ollama concurrency defect. LiteLLM was never independently ruled out, leaving another suspect in the kitchen.

Completion follows a chain of decisions. The runtime selects a model path and allocates cache space. Its scheduler interleaves live requests while the backend runs kernels chosen for the hardware and configuration. A model may finish perfectly alone, then expose an EOS or synchronization bug when another sequence shares its decode loop. A gateway can further transform or relay requests, obscuring the failure. Credible testing requires direct API calls, repeated concurrent runs and verification that every response terminates correctly. Then I care about tokens per second.

vLLM has hardware-specific bruises too. Reporting on a DeepSeek-V4.1-Flash deployment with RTX PRO 6000 Blackwell hardware, Lei-010 wrote:

Eager-mode decode on SM120 is effectively unusable for interactive serving of this model.

The report covered one model, the Blackwell architecture and a specific tensor-parallel configuration. It shows how nominal GPU support can diverge from useful serving performance when CUDA graph capture breaks or remains unavailable.

Automatic backend selection also needs supervision. In one eight-H100 GLM-5.3-Flash deployment, forcing FLASH_ATTN_MLA_SPARSE beat the automatically selected FLASHINFER_MLA_SPARSE_SM90 in reported serving benchmarks. Speculative-draft acceptance remained effectively unchanged, and the challenger attributed the difference to per-step attention cost. Automation is lovely until production makes you open the hood with olive oil already staining your shirt.

Put the Raspberry Pi at the front door

A Raspberry Pi makes a nice home server when it coordinates requests and leaves inference to a stronger machine. I’d run a reverse proxy there, maintain a small queue and wake the GPU box when prompts arrive. The workstation keeps the model warm and handles matrix operations. Splitting those jobs lets expensive hardware sleep during quiet hours. Ollama fits an occasional household workload with one or two active users. If agents overlap all day, serving moves to vLLM on the GPU host.

The NAS sits behind both machines, storing model files and snapshots while the compute server loads active weights into local memory. Network storage may slow startup, though available evidence provides no useful benchmark for that cost. I keep frequent models on fast local storage and use the NAS as the library: bellissimo filing cabinet, zero tokens.

TrueNAS is simplifying that library’s automation with its official Proxmox plugin. The plugin requires the October 2025 TrueNAS release or newer because earlier versions lack its versioned JSON-RPC WebSocket API. With TrueNAS on a separate machine, iSCSI works from Proxmox VE 8.x onward; NVMe/TCP requires the 9.x line. TrueNAS says ISO files and backups still need NFS or SMB because Proxmox cannot place them on those block transports. Clones, moves and imports remain host-side.

TrueNAS describes the workflow beautifully:

You click the button. TrueNAS builds the disks.

For now, I’d treat the plugin as early-adopter infrastructure. TrueNAS says enterprise support remains under active validation. An open issue also reports HA Manager refusing to restart stopped VMs because it considered the plugin storage type unsupported. That report involved a three-node Proxmox cluster running plugin version 2.1.23-beta2, establishing a specific compatibility gap rather than universal failure. A home lab can absorb midnight detective work; a production customer usually sends an invoice.

Before 2028 ends, serious local AI setups will resemble tiny restaurants: Raspberry Pi at the door, shared storage out back, GPU server cooking only when orders arrive. Buy the RTX 5090. Just hire the scheduler before the dining room fills.

Frequently asked questions

What is the difference between Ollama and vLLM?

Ollama is suited to private desktop use and low concurrency because it simplifies GGUF models and Apple Silicon. vLLM is suited to shared GPU endpoints because its cache sizing, batching, and sampling design allow more active sequences to share decoding steps when requests overlap.

Should I use Ollama or vLLM on an RTX 5090?

There is no controlled RTX 5090 comparison that holds model weights, quantization, context, request mix, concurrency and quality constant. The article favors vLLM for sustained concurrent GPU serving and Ollama for one-person home chat until queues become annoying in practice.

Can a Raspberry Pi be used as a home server for local AI?

A Raspberry Pi works best as a home server that runs a reverse proxy, maintains a small queue and wakes a stronger GPU machine for inference. The compute host keeps the model warm, while shared storage holds model files and snapshots.