What our local AI actually runs at
Measured throughput for the machines behind this blog: two gpt-oss models on an M3 Max, and why the 16GB GPU is not benchmarked for language work at all.
This page reports what the machines behind this blog actually do, measured rather than estimated. It is re-run monthly and the numbers below are dated. Everything here describes a setup that runs in production every night — not a rig assembled to produce an article.
Last run: 2026-08-24. 3 repeats per measurement, median reported.
The two machines, and why only one is benchmarked here
There are two, and they do different jobs. An M3 Max with 128 GB of unified memory runs the language models. An RTX 5060 Ti with 16 GB runs image generation, and nothing else.
That division is the first useful finding, and it is not a compromise — it is what a single consumer GPU forces on you. At the time of this run the image stack held 15,630 MiB of the card's 16,311 MiB, leaving 211 MiB free. There is no language model that runs in 211 MiB. On an earlier attempt, with rather more of the card free, Ollama was left with 150 MB of VRAM and a 20-billion-parameter model fell back to the CPU entirely — it produced 0.96 tokens per second, which measures a disaster rather than a GPU.
So the card is not benchmarked for language work here, and that is the honest answer rather than a missing row. Publishing a tokens-per-second figure for a GPU that will never serve a language model in this setup would be describing a configuration that does not exist. If you are budgeting for one consumer GPU and expecting it to do both jobs, this is the constraint nobody writes down.
Language model throughput
gpt-oss:20b on the M3 Max 128GB
20.9B parameters, MXFP4 quantisation, 13.8 GB on disk, 13.09 GB resident — fully on the GPU.
| Prompt | Generation | Prompt processing | Time to first token |
|---|---|---|---|
| Short (91 tokens) | 73.50 tok/s | 515.54 tok/s | 2.83 s |
| Medium (145 tokens) | 73.74 tok/s | 756.08 tok/s | 3.68 s |
| Long (1,576 tokens) | 72.43 tok/s | 1259.38 tok/s | 4.61 s |
gpt-oss:120b on the M3 Max 128GB
116.8B parameters, MXFP4 quantisation, 65.4 GB on disk, 64.68 GB resident — the whole model fits in memory.
| Prompt | Generation | Prompt processing | Time to first token |
|---|---|---|---|
| Short (91 tokens) | 51.36 tok/s | 162.95 tok/s | 3.60 s |
| Medium (145 tokens) | 50.54 tok/s | 215.35 tok/s | 5.76 s |
| Long (1,576 tokens) | 46.68 tok/s | 597.80 tok/s | 8.25 s |
What the numbers say
Generation speed barely moves with prompt length. Each model sits near the same figure whether the prompt is ninety tokens or sixteen hundred, because generation is bound by memory bandwidth rather than by how much there is to read. Prompt processing does the opposite and climbs sharply with length, because a longer prompt parallelises better. If you are choosing hardware for long-context work, the second number is the one that changes your day.
The more interesting comparison is between the two models. The 120B model has 5.6 times the parameters of the 20B but runs only about 30 per cent slower — 51.4 against 73.5 tokens per second on a short prompt. That is what unified memory buys: as long as the model fits, more parameters cost far less speed than the size difference suggests. The catch is in the word fits. The 120B model occupies 64.68 GB, which is more memory than most consumer graphics cards have in total, and roughly 4 times what the RTX card upstairs could hold even if it were doing nothing else.
Time to first token is the figure that flatters nobody. Both models take seconds, not milliseconds, on a cold cache — 4.6 s and 8.2 s on the long prompt. A warm repeat is much faster, but the first request is the one a user actually waits for.
How these were measured
- Counted, not estimated. Measurements come from Ollama's native generate endpoint, which reports token counts and durations directly. Tokens per second is arithmetic on those, never wall-clock divided by a tokeniser's guess.
- Time to first token is the first token of any kind, including the model's reasoning. Waiting for the first visible word measures something else.
- Cold cache. Every run uses a unique prefix, because the prompt cache will otherwise be mistaken for throughput — with it reused, one run reported almost twenty-six thousand tokens per second of prompt processing, which was the cache being read back rather than the machine working.
- Warm-up discarded, then the median of the remaining runs. Not the best of them.
- Deterministic: temperature zero, fixed seed, fixed generation limit.
- Contention checked. A run is discarded if image generation starts on the same machine, and the share of the model actually resident in memory is recorded rather than assumed. Both models above were 100% resident, with no discarded runs.
Changelog
2026-08-24 — First published. Two models measured on the M3 Max: gpt-oss:20b and gpt-oss:120b. The RTX 5060 Ti is not measured for language work because it is fully committed to image generation; that is recorded above rather than left blank.
Each monthly re-run is added here, so a change in the numbers is visible rather than silently overwritten.
Measured 2026-08-24 on hardware in Los Angeles. Re-run monthly. If a figure here looks wrong, it probably is — tell me and I will re-measure it.