The Model That Returned Nothing
While building my quiz pipeline (previous post), the topic-classification stage needed a small local model to read a chunk of documentation and pick one to three tags from a fixed taxonomy of about 80 topics. Straightforward classification, thousands of chunks, running free on an...
The newer, smarter model returned an empty string. Every time.
While building my quiz pipeline (previous post), the topic-classification stage needed a small local model to read a chunk of documentation and pick one to three tags from a fixed taxonomy of about 80 topics. Straightforward classification, thousands of chunks, running free on an old laptop.
I started with Qwen 2.5 3B, and it worked. Then came the reasonable-sounding upgrade: Qwen 3 is newer and benchmarks better, so surely use that.
Qwen 3 returned ''. An empty string. Not an error, not malformed JSON, not a
wrong answer. Nothing, on every single production prompt, while happily
answering simplified test prompts. That contrast (works on the toy prompt,
returns nothing on the real one) is the fingerprint of this whole bug class,
and it is worth six minutes of your time because reasoning models are becoming
the default everywhere.
The autopsy: the tokens were spent before the answer started
Qwen 3 is a reasoning model: before answering, it emits internal
<think>...</think> deliberation. Local runtimes cap output with a token
budget (num_predict in Ollama). Put those together against a production
prompt carrying an 80-item taxonomy plus instructions plus the chunk:
The model was not failing to classify. It was thinking at length about a long taxonomy, burning the entire output budget on deliberation, and the budget ran dry before the first character of actual JSON. The simplified prompt needed less thinking, so it finished in budget, which is exactly what made the bug so misleading: every quick sanity check passed.
The diagnosis came from a small script that ran the SAME model against the
production prompt and progressively simplified variants, watching where output
died. (Qwen 3 has a /no_think toggle; in my testing it was unreliable
precisely on complex production prompts, the only place I needed it.)
The rule I extracted: match the model type to the task type
Classification wants non-reasoning, instruction-tuned models. Pick from a list, label sentiment, detect intent: deliberation adds latency, cost, and this failure mode, while adding nothing to a task with no reasoning in it. Boring instruction-tuned models (Qwen 2.5, Llama 3.2 class) answer directly and reliably. My classification stage ran on Qwen 2.5 3B across 1,406 chunks with zero errors, on hardware that struggles to open a browser.
Reasoning models belong where reasoning IS the value: multi-step extraction, code analysis, math, planning. Paying the thinking tax there buys you something.
Two operational corollaries, learned with real (small) money:
- If you must run a reasoning model, budget for the thinking. Output caps need to cover deliberation PLUS answer (thousands of tokens, not hundreds), and on paid APIs the reasoning tokens bill as output: my cost estimates ran 3 to 4 times over the base-rate math until I calibrated against actual burn instead of price-sheet arithmetic.
- An empty response is a diagnostic, not a mood. When a model returns nothing on the real prompt but works on the toy one, suspect the token budget and the hidden deliberation first, before rewriting the prompt for the fifth time.
The wider point
"Newer and smarter" is not an upgrade path, it is a different tool. Model families now split by COGNITIVE STYLE, not just size and recency, and the benchmark-winning style can be structurally wrong for your task. The five minutes it takes to ask "does this task actually need reasoning?" is the cheapest evaluation you will ever run.
Pictures to add later
(none needed: this one is all terminal and ideas. Optional: the diagnose script's output showing empty vs valid responses side by side)