3,736 Tokens for Five Words: Thinking Is a Role, Not a Default

2026-11-01 · 6 min read · by Michael Kaminski

I asked a 35B mixture-of-experts model running on my desk, qwen3.6:35b-a3b under Ollama on an M5 Pro, to answer a question in five words. Thinking off: 122 ms. Thinking on: 36.1 seconds and 3,736 tokens, for five words. Measured 2026-11-01: 267x to 296x on wall time, about 500x on tokens, the same 105 tok/s decode in both modes, 876 tok/s prefill, one serialized stream however many requests you send. Why the fleet I built on 2026-10-30 names thinking as a separate role with its own timeout and retry budget instead of a default, what that costs (a 40-line validator that cannot catch a well-formed wrong answer, 29 GB resident), and the admission that fleet.db has zero runs, so every number here is the engine on a stand.

I asked a 35-billion-parameter model running on my desk to answer a question in five words. With thinking off it took 122 milliseconds. With thinking on, the same model, same prompt, same machine, took 36.1 seconds and generated 3,736 tokens to arrive at five words.

The conclusion first: on a local model, reasoning mode is a 300x latency multiplier, and it belongs in a role a planner has to choose on purpose, never in a default.

I built a small agent fleet on 2026-10-30 with exactly that rule in its config. Everything below was measured on 2026-11-01 on the machine that runs it.

The fleet, in one paragraph

fleet is 1,332 lines of Python across eleven files, standard library only, plus a 200-line test file with 17 tests. Claude plans a goal into tasks, a local Ollama model executes them, SQLite carries the run across days with lease-based claims, and a built-in dashboard reads the same handlers as the CLI and the MCP server.

Callers never name a model. They name one of five roles in fleet.toml: worker-bulk, worker-think, escalate, integrator, planner. The first two are the same local model, qwen3.6:35b-a3b under Ollama 0.40.7 on an M5 Pro with 64 GB. The only difference between them is one line: think = false versus think = true.

That one line is the subject of this essay.

Six words, 3,736 tokens

The prompt was "Answer in exactly five words: what does a dispatcher do?", sent to Ollama's /api/chat with stream: false, model already resident, two runs in each mode. A first cold call with thinking off took 1,470 ms, almost all of it prompt evaluation while the runner woke up; I discard it and report the warm runs.

The ratio is 267x to 296x on wall time and roughly 500x on tokens. The decode rate is the same in both modes, about 105 tokens a second. The model is doing five hundred times more work to reach an answer of the same length.

On quality: thinking on produced five words both times. Thinking off produced five words once and six the other time. On this prompt, that one extra word is the entire difference 3,736 tokens bought.

What 30 seconds does to a day

The fleet's working unit is a bulk task of about 8,000 tokens in and 1,000 out: a brief shared by every task in the run, the task itself, and a JSON answer. Two more measurements price that unit.

Prefill: a 10,520-token prompt took 12.0 seconds, which is 876 tokens a second. Decode: 105 tokens a second, from the long runs above. So one bulk task costs 8,000 / 876 + 1,000 / 105 = 9.1 + 9.5 = 18.7 seconds, and one stream clears 86,400 / 18.7 = about 4,600 tasks a day.

Add the thinking budget the model spent on a trivial question, 3,227 to 3,736 tokens, and the same task takes 49 to 54 seconds. The day drops to 1,590 to 1,750 tasks. On a question that actually needs multi-step reasoning the budget is larger, so that is a floor on the cost, not an estimate of it.

Roughly: thinking on by default would hand back two thirds of the machine.

One stream, however many you send

The number of streams matters because the thinking cost is paid serially. I sent one request for about 65 tokens of output and it returned in 0.83 to 1.15 seconds. I then sent four at once.

The four came back at 0.94, 1.96, 3.15, and 3.95 seconds. Their decode times sum to 3,751 ms and the wall clock read 3,950 ms. Ollama's MLX runner on this machine serves one stream and queues the rest, which is why fleet.toml sets local_concurrency = 1 and OLLAMA_NUM_PARALLEL=1 in the launchd environment. Extra slots would only have hidden the queue.

So a 30-second thinking call is not a slow task. It is 30 seconds during which nothing else on the local tier can run.

Role, not model, and not default

The rule the fleet encodes is that a caller asks for a role and the config decides what a role means. The planner's system prompt says it in one line: use worker-bulk by default, worker-think only for multi-step reasoning, integrator only for a final merge. Every task carries a role name, and the dispatcher looks the role up at call time, so swapping a model, or turning thinking on for a class of work, is an edit to one file that the running worker picks up without a restart.

The two local roles also differ in budget. worker-bulk gets a 120-second timeout and two retries. worker-think gets 300 seconds and one retry, because a second 30-second think on a task that just failed one is a worse bet than escalating.

Escalation is the other half of the design. After the retries, the dispatcher walks the role's fallback one tier up, to a claude-haiku role over the Anthropic API, and marks the result escalated = 1 so the dashboard can show what the local tier could not do. At the rates in the config, $1 per million in and $5 per million out, a bulk-shaped escalation costs about $0.013. A planned run costs about $0.275, paid once, to Opus.

Thinking off by default and a paid tier above it is the whole bet: let the free model take the first swing at everything, cheaply, and pay for the second swing only on the tasks that earned it.

What it costs

Three things, and the third is the one that matters.

First, thinking off means a bulk task that quietly needed reasoning gets a confident shallow answer. The only gate behind the worker is a 40-line validator that checks a JSON-schema subset: type, required, properties, items, enum, string length. It catches a malformed answer. It cannot catch a well-formed wrong one, and a wrong one that validates never escalates. The planner's judgment about which tasks need worker-think is doing work that nothing downstream verifies.

Second, the memory. The model is 23 GB on disk and 29 GB resident at a 32K context when I ran ollama ps today, up from 27 GB earlier in the same session as the context cache grew. Keeping it resident forever (OLLAMA_KEEP_ALIVE=-1) is what makes the 122 ms possible, and it is 29 GB of a 64 GB machine that nothing else gets.

Third: fleet.db has zero runs, zero tasks, zero results, zero events. It was last written at 16:14 on 2026-10-30, the afternoon the schema was created, and nothing has been dispatched through it since. Every number in this essay is a measurement of the engine on a stand. The architecture doc's line about "$1,702 a month avoided" is 4,364 local tasks a day times a Haiku-equivalent price, which is capacity arithmetic for work that has not happened, and last week I wrote 1,500 words about the difference between a system that runs and a system that would.

The measured numbers stand. The dollar number does not, until a run does.

The rule

Treat reasoning mode on a local model as a separate, more expensive tier with its own name, timeout, and retry budget, and make the planner spend it deliberately. Measure the think-on and think-off latency on a trivial prompt before you set the default. If the ratio is in the hundreds, as it was here, the default is off, and "on" is a role.

If you are routing between a local model and an API tier and your config has a single thinking flag for the whole worker, I would like to see your think-on-versus-off numbers on the same prompt. Mine are above, and I would trade.