When you quantize a Mixture-of-Experts model to fit it on cheaper hardware, some tokens start picking the wrong experts. Top-k MoE routing is a discontinuous function: small numerical disturbances push tokens across decision boundaries, and MoE route flips are the result. A new arXiv paper measures how much quality loss flows through that routing channel, and the finding should make any team deploying quantized MoE models reconsider cheap inference-time fixes.
The paper, published August 14, 2026, studies OLMoE-1B-7B under 4-bit KV-cache quantization with a protected BF16 gate. The route-mediated fraction of quantization damage is RMF approximately 0.31, meaning roughly a third of the quality loss routes through expert selection changes rather than pure compute degradation. The deployable router margin can detect that a flip occurred at AUC 0.772. But telling a harmful flip from a helpful one scores at AUC 0.490, statistically indistinguishable from chance.
What exactly did the researchers measure?
The authors built a four-run causal apparatus that decomposes quantization damage into two paths: a compute path, where the quantized KV cache degrades attention, and a routing path, where the same quantized cache pushes tokens to different experts. The setting is OLMoE-1B-7B, an open model with 16 layers, 64 experts, and top-8 routing. The router gate stays in BF16 to isolate the KV disturbance from gate-level quantization noise.
The headline number is the route-mediated fraction. On OLMoE at 4-bit KV, RMF approximately 0.31 with a 95% confidence interval of [0.20, 0.41], excluding zero. The same paper replicated this across five independent process invocations, yielding a mean of 0.313 plus or minus 0.020 and a range of 0.288 to 0.339. A pre-registered re-execution on a held-out split gave 0.231, and fixed-process re-estimates ranged from approximately 0.23 to 0.33. The spread across methods is real, but the signal holds: routing changes account for a meaningful share of quantization damage.
The authors also decomposed the damage by mechanism. Roughly 45% localizes to a single scoring layer as a jump, 55% is nonlocal from upstream layers, and about 0.2% is pure-flux. Nearly all, 99.8%, of the net signed route-mediated contribution is associated with a route-set change at the scoring layer or upstream, with the majority being nonlocal. The disturbance is cascading, spreading across the full depth of the model.
Why can a detector find flips but miss the harm?
This is where the paper moves from measurement to a sharper claim. The deployable router margin, a single inference-observable statistic, scores flip occurrence at AUC 0.772. That is a solid detection signal. When the same margin tries to predict whether a detected flip will increase or decrease the token's loss, the AUC drops to 0.490, which is chance. The conditional probability that a flip is harmful given that it occurred is 0.572, barely above a coin flip.
The three AUCs operate on distinct populations, shown in the chart below. Flip detection at 0.772 runs on all tokens. Harm-given-flip at 0.490 runs only on flipped tokens. Benefit-versus-all at 0.499 runs on all tokens and tests whether the margin can pick out flips that help. Neither of the last two beats a rate-matched random selector.
The benefit predictor reconstructs as a mixture over two negative groups: non-flip tokens, separable at AUC 0.615, and flipped-but-harmless tokens, separable at AUC 0.342. The harmful flips hide inside the harmless ones, and the margin cannot separate them. The authors extended the feature family to a cross-layer router vector spanning all 16 layers. It still scored at chance. The barrier persists across the full router stack.
The intuition for why flips go both ways is visible in the signed contributions. Harmful flips add 0.043 nats per token. Beneficial flips subtract 0.047 nats. The net is approximately zero, which is why the route-mediated fraction is reported as a signed set-change share rather than an absolute-value partition. Quantization noise does not uniformly break routing. It reshuffles it, and the reshuffling sometimes helps.
What does this mean for teams shipping quantized MoE models?
If you are quantizing an MoE model for inference, expect about a third of your quality loss to come from routing changes, not from degraded attention computation. That has concrete implications for mitigation.
- Selective repair is bounded at chance. Any inference-time scheme that detects flips and repairs only the harmful ones, using the local router statistics tested here, inherits the barrier. Online token-wise router calibration built from those features will not beat random selection. If you are building a flip detector that patches only bad routes, it will not work with these features.
- Full-reference repair pays out, but modestly. Pinning a clean route recovers a bounded slice of the routed ceiling: 0.23 to 0.46 nats on two architectures. The payout is architecture-modulated, so your results will vary by model. A controlled same-checkpoint flag-swap re-scoped the gate's normalization convention from a route-recoverability mechanism to a damage-magnitude moderator, so do not count on normalization tricks to save you.
- The damage is cascading, not local. With 55% of the damage from upstream layers and only 45% from the scoring layer, a single-layer fix addresses less than half the routing damage. Multi-layer intervention is more expensive and harder to justify when the per-token benefit is unpredictable.
For cost modeling, the cheap path is closed. Selective detect-and-repair at inference does not work with the tested features. Full-reference repair or avoiding KV quantization is what remains. The paper proposes no new mitigation. It supplies a measurement and a bound.
Does this rule out fixing routing at inference time?
The barrier is empirical, scoped to the tested inference-observable router statistics at pilot scale on the architectures measured. The authors are explicit: the paper does not rule out predictors that use richer hidden-state information or trained-decoder features. It also does not claim that quantization is uniquely harmful to MoE, or that route repair is impossible in general.
What it does say is that the obvious, cheap approach fails. Look at the router margin, flag suspicious flips, fix only those. The information observable at inference time from the router alone cannot distinguish good flips from bad ones.
Related work reinforces that routing is fixable with the right information. A separate study on expert misrouting in MoE models found that a minimal router-only update to the final-layer router, leaving every expert frozen, is sufficient to shift pass@K on AIME 2024 and 2025 and HMMT 2025 for Qwen3-30B-A3B and GPT-OSS-20B. Another paper on counterfactual routing for hallucination mitigation showed that a method called CoR improves factual accuracy by 3.1% on average without increasing the inference budget. These approaches work because they use training or counterfactual information, not just the local router margin at inference time.
The distinction matters for build decisions. If you are working with large MoE models like Kimi K3's 2.8T-parameter architecture or planning quantized deployment, the detection half of your routing pipeline can work. The triage half, at least with features observable without a reference model, cannot. Our earlier coverage of quantized MoE route flip detection and repair tracks the broader landscape, and this paper sharpens the picture.
The detection ceiling
The finding that should stick: AUC 0.772 for detecting a route flip, AUC 0.490 for predicting whether it hurts. The gap between those two numbers is the story. You can build a monitor that tells you something went wrong. You cannot build a cheap repair loop that fixes only the damage. For teams deploying quantized MoE models, the practical takeaway is to budget for full-reference approaches or accept the routing loss. The selective-repair shortcut, for the features observable at inference, is closed.
Sources
- arXiv: Detecting a Route Flip Is Easier Than Knowing Whether to Fix It: Causal Route-Mediated Damage in Quantized Mixture-of-Experts
- arXiv: When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
- arXiv: Awakening Dormant Experts: Counterfactual Routing to Mitigate MoE Hallucinations
