The best local LLM in 2026 — three models, three jobs

Three local models win different jobs: Qwen for interactive coding, Gemma for agent loops, and gpt-oss:20b…

The short version

The best local LLM depends on workload: Qwen for coding, Gemma for agents, and gpt-oss:20b for Apple Silicon.
Gemma reached 199 tokens per second, while gpt-oss:20b reached 74 on an M3 Max.
Benchmark complete tasks, latency, failures, context, and runtime versions before spending money on more GPU hardware.

The best local LLM in 2026 is three models: Qwen3.8-27B for interactive coding, Gemma 26B-A4B for unattended agents, and gpt-oss:20b as my Apple Silicon default.

Martin Merz described the requirements for European digital resilience:

La résilience numérique de l’Europe repose sur une souveraineté à la fois sécurisée et scalable.

Expecting one checkpoint to dominate every local workload is like asking one knife to run an Italian kitchen. The cleaver can trim artichokes, technically. Nonna will still remove you from the premises.

I rank models by useful work completed after memory limits and failures enter the bill. Benchmark wins matter less when a deployment spends Tuesday hallucinating JSON or borrowing half the system RAM.

Start with the machine, then choose the model

A local model earns my recommendation when its working configuration fits comfortably. Weights consume memory, but the runtime also needs room for its KV cache and temporary allocations. Longer conversations enlarge that cache. A checkpoint that survives a short prompt may spill during a repository session, sending transfers across a slower bus and wrecking the sexy GitHub performance profile. Sparse mixture-of-experts models use less computation per token, although their wider collection of weights must remain accessible. I leave headroom because computers sometimes perform exotic secondary duties like running an operating system. “It fits if nothing else is open” means it does not fit.

Bar chart comparing current figures against their baselines: dual-3090 DFlash2 greedy decode… 474 tok/s versus 432 tok/s, single-stream prose decode on the… 85 tok/s versus 98 tok/s, single-user speculative decode at long… 32 tok/s versus 68 tok/s, steady-state aggregate decode throughput… 1100 tok/s versus 46 tok/s.

Flash-Next shows why architecture complicates shopping. Gated DeltaNet compresses most running history, while Qwen Sparse Attention recovers relevant long-range context by scoring selected micro-blocks rather than every previous token. Together, they reduce the work needed to retrieve useful information from long prompts. Gated residual reads control how information returns from widened residual streams, preserving expressiveness with little inference overhead. Qwen also stores capacity in an n-gram embedding table that the runtime can prefetch from host memory instead of keeping entirely on the accelerator. Its sparse design activates 6B parameters per token, down from 17B in the Qwen3.7-Plus predecessor. Clever architecture; your runtime decides whether the cleverness reaches the chair.

Karine Brunet emphasized the realities of interconnected technology ecosystems:

Today’s organisations operate in highly interconnected technology ecosystems where complete independence is rarely achievable.

My setup makes this painfully literal. The RTX 5060 Ti with 16GB handles image generation, while ComfyUI leaves Ollama roughly 150MB of VRAM. A 20B language model therefore runs entirely on the CPU. The GPU appears in the machine description and contributes nothing to language inference, like the cousin who arrives after dinner and asks whether he can help.

Technician holds a second RTX 3090 above a dual-card workstation for local LLM testing at a computer-parts shop.

The same trap appears at the expensive end. SGLang documents the Flash-Next NVFP4 checkpoint at 126 GiB, while one DGX Spark has 128GB of unified memory. Conventional loading still fails because pinned host allocations use that shared pool. SGLang’s guidance calls for two connected Sparks or moving the embedding table to file-backed NVMe storage. Fine engineering. Expensive aperitivo. Anyone downloading the model onto one Spark tonight has acquired a weekend infrastructure project.

Parameter count does not predict speed. A sparse model may fly on a mature backend and crawl elsewhere when routing or memory access hits an ugly kernel path. One AMD Strix Halo report found Flash-Next decoding became about four times slower at longer context than with the reporter’s short prompt, although output quality remained intact. Runtime maturity can mug beautiful architecture in the parking lot.

Qwen for the desk, Gemma for the night shift

For interactive coding, I pick Qwen3.8-27B on a GPU with 32GB of memory. I must read the explanation, inspect the patch and choose the next prompt, so a weak answer wastes my attention even at Formula One speed. The first serious attempt must be worth reviewing. I use the tested memory tier rather than squeezing the model onto a smaller card and pretending offload preserves the comparison. A benchmark describes one setup; vibes around the model name do not survive a PCIe bottleneck. My attention costs more than the machine time saved by a faster mediocre patch.

In the reported RTX 5090 benchmark, the speculation-enabled preset generated 85 tokens per second versus 74 without speculation under the same configuration. Both are comfortably interactive. The benchmark author chose Qwen for coding and thinking because its behavior on those tasks justified the tighter fit.

Gemma 26B-A4B gets my background queue. On the same author’s agent suite, it matched the reported dense alternatives while decoding at 199 tokens per second with the tested Q4_K_M setup. That earned Gemma the unattended-loop job.

The mechanism matters because unattended agents often pursue doomed plans. Gemma is a mixture-of-experts model, so its router sends each token through a smaller active portion of the resident network. A backend that routes efficiently can reduce weight traffic during generation. Faster decoding shortens successful and failed attempts. Cheaper failures make retries affordable, letting me enforce hard timeouts without occupying the machine all afternoon. Dense alternatives achieved similar task results in the reported suite, but their slower loops charged more machine time per dead end. On scoped edits with repeated tool calls, that cost can determine which model completes more useful work overnight.

I used to leave agents running because stopping felt wasteful. Founder brain loves the sunk-cost fallacy; we rename it “persistence” and put it on a slide. Hard timeouts fixed more of my agent infrastructure than another round of prompt poetry.

Speculative decoding deserves similar suspicion. A smaller prediction path proposes several tokens, which the target model verifies before output. Accepted drafts can accelerate generation while preserving the target model’s decisions in the intended design; rejected drafts add work. On vLLM’s documented H100 configuration, multi-token prediction reduced request throughput and increased per-token latency because acceptance was too low. Hardware and context decide whether the drafter earns its electricity. I benchmark speculation per deployment and disable it when the receipt gets ugly.

Reliability adds another wrinkle. A greedy SGLang comparison with Qwen thinking enabled produced a deterministic token-sequence divergence between DFlash2 and the target-only run, while the no-thinking control matched exactly. That result covers only that implementation path, but it challenges automatic claims of identical output. For coding agents, I test the exact thinking mode and runtime I intend to ship.

gpt-oss:20b is my Apple Silicon default

On my M3 Max with 128GB of unified memory, gpt-oss:20b generated 74 tokens per second while fully resident. I measured the MXFP4 checkpoint on August 25.

The larger gpt-oss:120b managed 51 tokens per second on the same machine. The smaller model was roughly half again as fast.

Prompt ingestion widened the gap: 756 prompt tokens per second for the 20B model versus 215 for its larger sibling. In document-heavy chats, that controls how long I stare at an idle window before generation starts.

The larger checkpoint still ran surprisingly well. Its first token took about six seconds, versus roughly four for the smaller model. I can tolerate that when the larger model proves better at a specific job, but my measurements lacked an identical independent quality suite. Two speed counters cannot manufacture a quality verdict, however much YouTube-thumbnail energy I apply.

That uncertainty makes gpt-oss:20b my default installation, not an all-purpose champion. It starts sooner, ingests prompts much faster and leaves ample unified memory on this machine, making experimentation painless. If a larger model wins on my tasks, I can promote it with evidence. Starting with the heavyweight because its parameter count looks impressive turns an expensive Mac into a space heater with excellent typography. Local workflows also refuse to stay isolated: the browser remains open, an IDE indexes the repository, Docker chooses violence against the filesystem, and an image pipeline may already own the discrete GPU. A model that leaves room for everything else often completes more work than an empty-machine benchmark winner. I want a setup I can leave running, close the laptop and trust.

Benchmark your workload before buying another GPU

Local inference numbers are spectacularly easy to misread. A controlled dual-3090 test found tensor parallelism delivered 16% to 35% more greedy-decode throughput than one RTX 3090 on the same harness. The pair communicated over PCIe, so the second card helped without approaching double performance.

More cards can hurt an individual user. In a four-card test, one prose stream decoded at 85 tokens per second, versus about 98 on an NVLinked two-card pair. Four-way all-reduce crossed PCIe between the pairs. The larger setup held far more context, but one answer arrived more slowly.

Concurrency changes the result again. One RTX 3090 batch-mode test reached about 1,100 aggregate tokens per second with 64 concurrent requests, versus 46 with one request. Useful for a shared server; nearly useless for judging one coding response’s latency. A benchmark can be accurate and answer the wrong purchasing question.

Long context sends its own invoice. On one RTX 3090, a compressed KV mode expanded the available pool from roughly 174K to 292K tokens. With the tested long prompts, speculative decoding fell from 68 to 32 tokens per second compared with the FP8 cache. The extra context worked. Generation paid.

Binary changes can overwhelm hardware comparisons. In one controlled Ollama report, an update cut Qwen3-VL generation by about 80%, dropping the same workload from roughly 129 tokens per second to 26. The physical cards were swapped between runs, yet performance followed the software version. Before buying another GPU, I pin the runtime and keep the old binary nearby. I have spent enough nights blaming silicon for regressions introduced three folders higher in the stack.

My local test uses fresh prompts from recurring work. I record time to first token and total completion time, then judge whether the patch or tool call was usable. Every candidate gets a hard timeout. When measuring cold behavior, I restart long-lived servers because automatic prefix caching can recognize token-identical synthetic prompts, reuse previous work and distort repeated sweeps as cache entries accumulate. If one model finishes two jobs cleanly while another produces five fast disasters, the arithmetic is mercifully simple.

Nobody has published the independent cross-hardware evaluation I want: leading local models running identical agent workloads with the same reliability checks. We also lack dependable boundaries for when multi-token prediction improves user-visible latency. Browser and Windows NPU runtimes have not yet proved equivalent tool behavior for the same checkpoint.

By September 2027, I expect “best local LLM” to sound as incomplete as “best employee.” Any model wanting Qwen’s desk, Gemma’s night shift or the default slot on my Mac can apply by finishing the work before the timeout.

Frequently asked questions

What is the best local LLM in 2026?

The best local LLM depends on the workload and hardware. Qwen3.8-27B is the interactive coding pick on a 32GB GPU, Gemma 26B-A4B suits unattended agent loops, and gpt-oss:20b is the default for Apple Silicon because it starts sooner and ingests prompts quickly.

How much memory does a local LLM need?

Model weights, the KV cache, temporary allocations, context length, and runtime overhead must fit comfortably in available memory. A model that barely loads can spill during longer sessions, slow across the system bus, and crowd out the operating system, browser, IDE, Docker, or image pipeline.

How should local LLM performance be benchmarked?

A useful local LLM benchmark measures fresh prompts from recurring work, time to first token, total completion time, and whether the output is usable. Runtime versions should be pinned, candidates should receive hard timeouts, and long-lived servers should restart for cold tests to prevent prefix caching from distorting results.