If you serve a language model with a million-token context window, every single token the model generates requires scanning the entire conversation history. For a 1M-token context, that means loading roughly 15 GB of key-value cache data at each decoding step. The model already knows which few tokens matter. The attention mechanism has no way to act on that knowledge, so it reads everything, every time.
A paper from researchers at KAIST AI and Google DeepMind, published September 2 on arXiv, proposes a protocol called Declarative Attention that turns this bottleneck into a prompting problem. The model declares, inside its own chain-of-thought, which region of the context it needs to read next. The inference engine parses that declaration like a tool call and skips the rest of the KV cache. On Gemma-4-31B, the protocol cut attended tokens by 52.0% with an accuracy drop of just 1.27 percentage points across 15 long-context benchmarks.
How does Declarative Attention actually work?
The protocol partitions generation into three modes that the model toggles between using XML-like tags in its output. Global mode gives the model full access to the entire context, used typically at the start of a task to survey the document or identify key sections. Focus mode restricts attention to a specific labeled segment, say chunk K, so the model can reason over one paragraph without loading the surrounding text. Local mode ignores external context entirely and works only from the prompt instructions and the model's own generated reasoning so far.
The context gets split into semantic units the authors call "magic chunks," each roughly 2,000 tokens long. A scaffold containing the system instructions and the user's question stays visible in all three modes, acting as a persistent reference point. The model receives a structured prompt that defines the tags and explains when to use them. No fine-tuning is required.
The system integrates with existing efficient attention kernels through the vLLM serving framework. Rather than masking individual tokens, which would be inefficient on modern GPUs, the engine operates at the level of KV-cache blocks, typically 16 to 32 tokens at a time. When a segment is masked out, the hardware skips loading those blocks from memory entirely. The speedup comes from reducing global memory reads, which are the primary bottleneck in modern LLM inference on GPUs.
The closest existing approach uses lightweight proxy scores to pre-select relevant tokens before running full attention. Those methods still scan the entire context at every step to compute the proxy, incurring O(N) cost per token generated. Declarative Attention sidesteps that scan because the model itself has already identified what it needs through its own reasoning. The proxy is the model's chain-of-thought.
What do the benchmarks actually show?
The researchers evaluated DA zero-shot across 15 long-context tasks on two off-the-shelf models: Gemma-4-31B and Qwen-3.6-27B. No fine-tuning, no specialized training, just a structured prompt.
The results split cleanly along model scale. Gemma-4-31B reduced total attended tokens during decoding by 52.0%, with an accuracy drop of 1.27 percentage points. Qwen-3.6-27B saw a 31.1% reduction with a 2.75pp drop. The larger model both cut more tokens and lost less accuracy.
The chart above shows the tradeoff for both models side by side. The gap between the token reduction bars and the accuracy drop bars is the whole story. You are removing half the computational work and losing roughly one percentage point of accuracy on the larger model. For Qwen, the tradeoff is less favorable but still clearly positive.
An important detail: the accuracy costs shrink with model scale. The 31B model retained 99% of the vanilla model's accuracy under DA. The smallest model tested, at 4B parameters, retained only 29%. Controlling your own attention appears to be a capability that emerges with scale, much like chain-of-thought reasoning itself. This is a technique for capable models, not a free lunch for small ones.
How much does this save in wall-clock time?
Token reduction is a proxy for what builders actually care about: latency and cost per request. The researchers used a roofline analysis, a method for estimating the maximum achievable speed of an algorithm on specific hardware, to project real decode times.
For Gemma-4-31B, DA reduced decode wall-clock time to 0.71x of the original, meaning a 29% speedup. For Qwen-3.6-27B, the projected decode time was 0.77x, a 23% improvement. These gains come from reducing global memory reads. During decoding, the model is memory-bandwidth bound, not compute bound. Skipping KV-cache blocks means less data movement, and less data movement means faster token generation.
To put that in dollar terms: if you are serving a 31B model with long contexts, a 29% reduction in decode time translates directly to roughly 29% fewer GPU-seconds per request. On an H100 at current cloud rates, that adds up fast at scale. It pairs well with other inference optimizations like GPU scheduling order improvements that recover performance without new hardware.
Where does the protocol fall short?
The zero-shot nature of DA is both its biggest advantage and its biggest limitation. The fact that it works without training means you can apply it to any compatible model today. But the authors are explicit that this is a lower bound on what is possible. They frame training-based methods, where models are fine-tuned to use attention declarations natively, as future work with significant potential.
The scaling dependency is a hard constraint. If your stack runs on a 4B or 9B model for cost reasons, DA will not work. The smallest models failed to adhere to the protocol reliably, and their accuracy collapsed to 29% of baseline. This technique is for teams running 27B and above, or for future models trained with DA from the start.
There are open questions about how DA interacts with agentic workflows. The protocol was tested on long-context question answering and retrieval tasks. Multi-turn agent loops, where the context grows dynamically with tool outputs and intermediate results, might produce declaration patterns the zero-shot prompt did not anticipate. If the model mis-declares and focuses on the wrong chunk, the error is silent. It simply does not read the region containing the answer, and you get a confident wrong response.
The roofline projections are estimates, not measured end-to-end serving benchmarks. The actual speedup in a production vLLM deployment with batching, speculative decoding, and concurrent requests may differ from the projected 0.71x. The token reductions are measured directly. The wall-clock gains are modeled.
What should builders do with this today?
If you are serving long-context models and care about inference cost, DA is worth prototyping. Here is what to consider:
- Model size gate. Only test DA on models of 27B parameters or larger. Smaller models lack the instruction-following reliability to use the protocol safely.
- Prompt engineering overhead. DA requires a structured system prompt defining the three modes and the chunk labeling scheme. Expect to tune the prompt for your specific use case, especially the chunk size and the scaffold format.
- Integration path. The implementation uses vLLM with block-level KV-cache masking. If you are already on vLLM, the integration surface is manageable. If you use a custom serving stack, you will need to implement the block-skipping logic yourself.
- Monitoring. Because mis-declarations are silent, you need logging on which mode the model selected and which chunks it focused on. A model that always selects global mode is getting no benefit, and you want to catch that before it shows up in your GPU bill.
- Evaluate before shipping. Run your own evaluation suite, not just the paper's 15 tasks. The average accuracy drop may be small but catastrophic on specific tasks that require cross-chunk synthesis or multi-hop reasoning.
This is also a signal for roadmap planning. If model vendors begin training models with DA-style declarations natively, the zero-shot accuracy penalties could disappear entirely. The paper's finding that larger models adhere better suggests this is a capability that training can amplify. A model trained to control its own attention would not need the structured prompt, and the accuracy retention would likely exceed the 99% the 31B model achieved zero-shot.
The real test is training, not prompting
Declarative Attention works zero-shot, which is remarkable. But the more interesting bet is what happens when a frontier lab trains a model to use it from scratch. The gap between 99% accuracy retention at 31B and 29% at 4B tells you this is a learned skill, not an architectural trick. Skills that emerge with scale tend to improve dramatically when you train for them directly. The 52% token reduction is a floor, not a ceiling.
For now, the paper gives builders a concrete protocol to try on models they already run. The implementation is open, the integration is vLLM, and the tradeoff is quantified. If you serve long contexts, try it. The gating factor is whether your model is big enough to handle the protocol.
Sources
- arXiv - Language Models Can Control Their Own Attention
- alphaxiv.org - Declarative Attention analysis and summary
- huggingface.co - Paper page on Hugging Face
