The standard deal in model compression has always been a trade: you give up some accuracy, and in return you get lower memory and compute costs. A new technique called Quantization-Aware Healing breaks that deal. Applied to OpenAI's GPT-OSS 120B model compressed to 60B parameters and then quantized to 4-bit MXFP4 precision, QAH produces a model that beats its own bfloat16 source on 7 of 9 benchmarks while using roughly 4 times less weight memory. The 4-bit student even matches the original 120B teacher on LiveCodeBench. The result is Hypernova-60B, an open-weight model released by Multiverse Computing under Apache 2.0, and a recipe that reframes quantization from a lossy tax into a second pass of teacher supervision.
What did QAH actually achieve on the benchmarks?
The pipeline starts with GPT-OSS 120B, structurally compresses it to 60B parameters using a tensor-network operator, recovers the compressed checkpoint in bfloat16 through distillation, and then re-quantizes to MXFP4 using QAH. The comparison that matters is the 4-bit QAH model against its own bfloat16 source, the best full-precision version of this smaller architecture that exists.
The 4-bit model wins on 7 of 9 benchmarks. The two where it trails, MMLU-Pro and SciCode, lose by margins of 0.2 and 1.4 points respectively, according to the arXiv paper. Everywhere else the quantized model is ahead, and the biggest gains land on the capabilities compression usually damages most: long-context reasoning, where AA-LCR jumps from 35.3 to 42.7 for a gain of 7.4 points, and math, where AIME 2025 rises from 70.7 to 76.3.
Against the original 120B teacher, the gap is narrower but still striking. The QAH student surpasses the full-size teacher on LiveCodeBench, scoring 66.5 versus 66.0, a difference the authors describe as within run-to-run noise. On GPQA Diamond it closes to within 1.6 points, 67.4 versus 69.0. The largest residual gap is on AA-LCR at minus 7.3 points, an extreme long-context benchmark where capacity lost to structural compression is intrinsically the hardest to recover.
One important caveat: the public Hypernova-60B release incorporates further training beyond the pipeline evaluated in the paper, so the published numbers reflect the authors' own measurements of the pipeline rather than the released checkpoint's final figures.
Why does healing from the original teacher work better?
The mechanism comes down to which model you distill from. Standard quantization-aware distillation, or QAD, works when the only change is quantization: the full-precision version of the exact same model exists and can serve as teacher. But once a model has gone through structural compression, fewer layers, heads, and neurons, that assumption breaks. The only candidate teacher is the recovered bfloat16 checkpoint, which is itself a distilled approximation of the original. Distilling from it anchors the quantized student to a degraded target and caps accuracy at the recovered checkpoint's ceiling.
QAH removes that ceiling with one change: it distills directly from the original, pre-compression model. The teacher is full-size at 120B. The student is 60B and running in MXFP4. They do not even share an architecture. But because a teacher's output distribution is architecture-agnostic, the size and shape mismatch does not prevent knowledge transfer. The loss is a straightforward KL divergence on output logits.
Crucially, the advantage comes from the teacher being uncompressed, not from a precision difference. The original GPT-OSS 120B release ships with its mixture-of-experts weights largely in MXFP4 already. So QAH distills a compressed 4-bit student from an uncompressed teacher of comparable precision, not from a higher-precision one. The quantization stage becomes a second, full pass of distillation against the original teacher, supervision that the bfloat16 checkpoint never received.
This matters for anyone building a compression pipeline. If you are compressing a model structurally and then quantizing, the choice of teacher for the recovery step is the single biggest lever on final quality. Getting it wrong caps your model at the accuracy of an already-degraded checkpoint.
How does QAH compare to QAT for training stability?
The head-to-head against quantization-aware training isolates the effect of the loss function from everything else. The researchers quantized a GPT-OSS 9B model to MXFP4 and tracked average performance across MMLU-Pro, LiveCodeBench, and GPQA Diamond as training progressed.
Both methods reach a similar peak: 54.9 for QAH versus 54.6 for QAT. The difference is in how they get there and what happens after. QAH reaches its peak in about 100 steps, roughly 7 times faster than QAT's 700 steps. Then it stays within about two points of that peak through all 1,200 steps of training. QAT collapses once past its peak, losing nearly 19 points by step 1,200, falling from 54.6 to roughly 36.
The chart below shows the divergence: QAH climbs fast and plateaus. QAT climbs slowly, peaks, and then falls off a cliff. The key reported values are QAH peaking at 54.9 around step 100 and holding, while QAT peaks at 54.6 around step 700 and then collapses to approximately 36 by step 1,200.
The practical consequence is a deployment risk difference. A QAT checkpoint needs careful early stopping against a held-out signal to avoid shipping a model that has already started to degrade. A sufficiently trained QAH checkpoint can be served safely because it simply does not drift. The mechanism is straightforward: KL distillation against a frozen teacher gives the student no incentive to move once it matches the teacher, while a cross-entropy task loss keeps pushing toward hard labels and eventually erodes capabilities the model inherited from the original.
For a team building production LLM pipelines, this is the difference between a recipe you can automate and one that requires a human watching a dashboard. The Hugging Face blog post describing the work calls this out explicitly as the reason the team adopted QAH over QAT.
What does this mean for your serving costs and hardware?
The accuracy story comes with a concrete efficiency story. At 4-bit precision the QAH model uses roughly 4 times less weight memory than the bfloat16 student. At half the teacher's parameter count, it roughly halves compute per token. For model families that ship in bfloat16 rather than 4-bit, like Qwen, the combined parameter and precision reduction would amount to roughly 8 times less compute per token.
What this means in practice:
- A 120B-class model's capabilities become deployable on hardware sized for a 60B model at 4-bit, a category of GPU that costs materially less to rent or buy.
- The healing recipe converges in hundreds of steps, not weeks of hyperparameter search. The paper explicitly aims for a recipe deployable without a multi-week tuning cycle.
- The QAH model does not require hand-tuned early stopping, which removes a class of operational risk from your deployment pipeline.
If you are currently serving a 70B or 120B model and paying for the full memory footprint, the QAH pipeline suggests you could serve a structurally compressed, 4-bit version that is at least as accurate on most benchmarks at a fraction of the cost. The open-weight Hypernova-60B release gives you a concrete artifact to test that claim against your own workloads.
What are the limits, and what should you watch?
Several caveats temper the results. The 4-bit model does not beat the bfloat16 source on every benchmark. MMLU-Pro and SciCode are losses, even if small ones. The largest gap against the original 120B teacher is on AA-LCR at minus 7.3 points, an extreme long-context benchmark where compression is intrinsically hardest to recover. If your workload depends on very long context windows, the capacity loss from structural compression remains a real cost.
The paper also reports a large and reproducible quality gap between distributed-training backends, which means the infrastructure you run QAH on affects the final model quality. This is an operational detail that gets insufficient attention in most compression research, and it is worth reading the full paper for the specifics.
The comparison the authors did not run is also worth noting. They did not test whether QAH helps when the only transformation is quantization without structural compression. The advantage comes specifically from the fact that the recovered checkpoint is itself a degraded teacher, and a structurally compressed model is the case where that degradation is worst. If your model is only quantized, not structurally compressed, standard QAD may be sufficient. Liquid AI's QAD work on their edge models, for instance, closes 48 to 73 percent of the quantization quality gap without the architectural mismatch that motivates QAH.
Concurrent work from NVIDIA on NVFP4 distillation reports similar QAT-style instability, which suggests the collapse problem is systemic rather than specific to this pipeline. Watch for whether QAH-style teacher selection becomes a standard part of compression toolkits, or whether the major labs bake something equivalent into their own post-training pipelines. For a deeper look at how far quantization can be pushed, our coverage of a two-bit LLM running in 60 MB traces the aggressive end of the same spectrum.
The recipe is the moat
The headline finding, that a 4-bit compressed model can beat its full-precision source, sounds like a benchmark trick. The mechanism is simple and reproducible: the quantization stage gets a second pass of distillation from a stronger teacher, and the bfloat16 checkpoint does not. Anyone running a compression pipeline can apply the same principle. The question is whether your team has the distillation infrastructure to do it, and whether your model is structurally compressed enough to need it. For most production deployments, the answer to both is trending toward yes.
Sources
- arXiv: Quantization-Aware Healing: A Practical Recipe for Recovering Compressed, 4-Bit LLMs
- Hugging Face Blog: Quantization-Aware Healing: a compressed, 4-bit model that outperforms its full-precision original
- Liquid AI: LFM2.5 Q4_0: Quantization-Aware Distillation for Edge Deployment
- arXiv: Quantization-Aware Distillation for NVFP4 Inference Accuracy Recovery
