Metacognitive Decoding: Efficient Inference for Diffusion Language Agents
Diffusion LMs emit tokens in parallel; metacognitive decoding spends compute and text only where needed for agent loops.
The explainer · 3-minute read · 272 words
The claim
A controller decides how much text a diffusion language model writes for each agent step.
The problem
Diffusion language models write many tokens at once, which suits fast agents.
Efficient interaction needs both right-sized computation and right-sized text.
Fixed decoding spends the same effort on every response.
How it works
The controller watches the current response need and the environment exchange.
It learns when to finalize tokens and how far generation should extend.
The controller is a lightweight plug-in on the diffusion backbone.
Training uses agent trajectories to balance correctness against generation cost.
The backbone itself stays unchanged.
Figures
What changes
Baselines cut cost by generating less text everywhere.
This method adapts generation to each response instead.
Across five benchmarks it cuts inference FLOPs by 20.6% on average versus the most efficient baseline.
Limits
The FLOP cut is an average across five benchmarks; per-task results vary.
Accuracy stays competitive, but the paper claims no uniform win.
The controller trains on agent trajectories, so new domains need new training.
Source · arXiv wire