Most LLM compression treats transformer blocks as independent parts. Score each block's importance, remove the lowest-scoring ones, and hope the model holds together. Multiverse Computing just published research that says this mean-field approach leaves enormous quality on the table when you cut deep, and they have a physics-inspired fix that changes the optimization entirely.
The paper, published on arXiv as 2602.00161, reformulates block selection as a constrained binary optimization problem that maps directly onto an Ising glass, a disordered spin system from statistical physics. At 50% compression of Llama-3.3-70B-Instruct, the method preserves MMLU near 77, while the strongest competing block-removal baseline falls to the mid-50s. That 23 percentage point gap is the difference between a model that stays useful and one that breaks.
The code is open source. The headline results require no retraining. And the method generalizes to heterogeneous architectures including mixture-of-experts models. For teams shipping compressed models to production, this is worth understanding before the next round of cost cutting.
Why is picking which blocks to delete so hard?
Block removal is the bluntest form of LLM compression: delete whole transformer blocks, reducing depth and parameter count in one step. The idea is simple. The optimization is brutal.
Most existing methods score each block independently, using magnitude, sensitivity, or "block influence" heuristics, then remove the lowest-scoring blocks. Another common shortcut removes a single consecutive run of blocks from the middle or end of the model. Both approaches assume blocks are independent, the way mean-field physics treats each spin as if it only feels an averaged background field from its neighbors.
But blocks are not independent. Whether removing block 20 hurts the model depends on whether you also removed block 19 or block 24. That interaction, or coupling, between decisions is exactly what mean-field methods throw away. As models get deeper and more heterogeneous, ignoring those couplings becomes expensive, especially when you want to remove a large fraction of blocks at once.
The combinatorial space is the other problem. If you have 80 blocks and want to remove 40, the number of possible configurations is C(80,40), roughly 10 to the 23rd power. Brute-forcing that is hopeless. But the structure of the problem, pairwise couplings with a fixed-size constraint, is precisely the regime where Ising solvers earn their keep.
How does the Ising glass formulation work?
The method, described in detail on Multiverse's Hugging Face blog, attaches a binary variable to each transformer block: 0 means keep it, 1 means remove it, like a spin pointing down or up. Then the team performs a second-order Taylor expansion of the model's loss with respect to those variables, producing an approximate Hessian matrix. The diagonal entries measure how much each block matters on its own. The off-diagonal entries capture the pairwise couplings between blocks, the many-body interactions that independent scoring ignores.
That reformulation turns "which blocks should I remove?" into an energy minimization problem: find the set of M blocks whose removal minimizes the energy, subject to removing exactly M of N blocks. Physically, this is an Ising glass with all-to-all couplings and fixed magnetization, where the fixed number of removed blocks plays the role of a fixed total spin.
The key empirical finding is that this energy is a strong proxy for downstream benchmark quality. Low-energy states of the spin system correspond to high-performing pruned models. You can rank billions of candidate configurations by a cheap energy calculation without ever running the actual model or benchmarking it.
The cost advantage is real. The Hessian is computed just once, from forward and backward passes on a small calibration dataset. After that, evaluating any candidate configuration is a single matrix operation. Because the couplings do not depend on the compression target, the same Hessian can be reused to solve for many different values of M. One calibration run gives you the full compression curve.
How much does this actually help?
The results scale with how aggressively you compress. At light compression, the method performs on par with state-of-the-art baselines across Llama-3.1-8B-Instruct, Qwen3-14B, and Llama-3.3-70B-Instruct. The couplings matter most when you are cutting deep, and that is where the gap opens.
For Llama-3.3-70B-Instruct evaluated without retraining, up to 24 of 80 blocks removed, CBO is roughly on par with block influence. But at 32/80 and 40/80, it pulls decisively ahead. At the deepest setting, 40 blocks removed out of 80, CBO holds MMLU near 77 while the strongest baseline falls to approximately 54, a gap of almost 23 percentage points. CBO beats the baseline on every benchmark tested at that compression level.
For Qwen3-14B at 12/40 blocks removed, CBO leads MMLU by about 10 percentage points. For Llama-3.1-8B-Instruct at compression rates exceeding 40%, the arXiv paper reports MMLU improvements of approximately five to seven percentage points over baselines. The pattern is consistent: deeper compression, bigger advantage.
The hardest exact case they solved was removing 8 of Llama-3.3-70B's 80 blocks, about 29 billion configurations, which took roughly two days of brute-force search on a single GPU. Beyond that scale, exact enumeration breaks down. But here the Ising formulation pays off a second time: cast as a QUBO (quadratic unconstrained binary optimization) problem, the same task can be handed to classical and quantum-inspired solvers. An open-source tabu solver reliably reaches the lowest-energy states in seconds, even on the hardest cases they could verify against brute force.
Does the best model come from the ground state?
Here is the subtlety that separates this from a standard optimization story. The energy is a strong proxy for quality, but not a perfect one. The single lowest-energy state is not always the best model.
This has a practical benefit. Once the Hamiltonian is set up, reading off the ground state and the low-lying excited states is essentially free, giving a spectrum of high-quality candidate prunings rather than one fragile answer. The authors found that for Llama-3.1-8B-Instruct at 16/32 blocks removed, most of the top 20 low-energy states cut blocks toward the end of the model, as prior work would expect. But the 17th excited state is the first to propose removing a block near the beginning of the model, and after light retraining, that configuration outperforms the ground state across several benchmarks.
That directly disproves the common assumption that the best pruning is one consecutive chunk of middle or late blocks. It also explains why lightweight solvers work so well: you need a handful of good low-energy states, and that is a much easier bar than finding the true ground state.
Does it transfer to non-standard architectures?
Modern models are not uniform stacks of identical transformer blocks. NVIDIA-Nemotron-3-Nano-30B-A3B-FP8, for example, interleaves Mamba2, attention, and mixture-of-experts layers in a non-uniform pattern. The Ising formulation does not care: a coupling is a coupling regardless of what kind of block sits at each site.
Applying the method to that hybrid model without any retraining, removing 2 or 3 MoE layers, CBO finds configurations that beat block influence on AIME25 and GPQA. Specifically, removing 2 MoE layers retains 94% of original accuracy on GPQA and 88% on AIME25. Removing 3 MoE layers retains 91% on GPQA and 74% on AIME25.
The chart above shows the retention numbers for the NVIDIA-Nemotron hybrid model. The results confirm that redundancy in these models is real but unevenly distributed: some expert layers are far more disposable than others, and searching the coupled configuration space is what locates the good cuts. The best configuration here is again an excited state, not the ground state.
The method also composes with the rest of a compression pipeline: quantization, low-rank and SVD compression, width pruning, and knowledge-distillation-based healing. Block removal is one stage, not a competitor to the others.
What should you do with this?
If you are shipping compressed models, here is what changes:
- Deep compression is viable without retraining. The 23-point MMLU gap at 50% compression means you can cut a 70B model to roughly half its depth and still have a model that scores 77 on MMLU. That is a serving cost story.
- The compute cost is front-loaded and small. One calibration run builds the Hessian, then any number of compression targets from the same matrix. A tabu solver runs in seconds. You can explore the full compression curve in an afternoon.
- It works on architectures you actually run. Mixture-of-experts models with interleaved attention and Mamba layers are the direction the frontier is moving. A pruning method that only works on homogeneous dense transformers is already obsolete.
- The code is open source. The repository on GitHub implements the method, and the Hessian-plus-solver pipeline is designed to apply to any architecture.
The caveats matter too. The headline 23-point result is on Llama-3.3-70B-Instruct at 50% compression without retraining. Lighter compression shows smaller gains, and the method is on par with baselines, not ahead. The energy proxy is strong but imperfect, which is why you want the low-energy spectrum rather than a single answer. And while the tabu solver works well on the sizes tested, the scaling story for models with hundreds of blocks is still empirical.
If LLM benchmark variance has taught builders anything, it is that benchmark scores move around. A pruning method that gives you a spectrum of candidate configurations, each benchmarkable independently, is more robust than one that gives you a single fragile answer.
The ground state is overrated
The most useful insight in this paper is the discovery that you do not need the optimal solution to get a great model. The whole low-energy spectrum works, and sometimes an excited state beats the ground state. That is a license to use cheap, fast, approximate solvers instead of chasing exact optima, and it is why a method rooted in quantum annealing theory can run on a laptop in seconds.
For builders, the practical takeaway is clear: block pruning just got a lot smarter, and the cost of trying it is an afternoon of calibration. The cost of not trying it, if your competitors are, is running a model twice as deep as you need.
Sources
- arXiv - LLM Compression by Block Removal with Constrained Binary Optimization
- huggingface.co - Pruning LLMs Like a Physicist: Block Removal as an Ising Optimization Problem
- arXiv - Block removal for large language models through constrained binary optimization
- runtimewire.com - Multiverse Computing's pruning method beats a baseline by nearly 23 MMLU points
- huggingface.co - Paper page for arXiv 2602.00161
- github.com - Block removal through constrained binary optimization, open source code
