The model on your laptop is not supposed to be the one running in the cloud. A project posted to Reddit this week argues that the gap between those two worlds is about to close from the bottom up. A developer trained a 250M parameter language model from scratch on 30B tokens of FineWeb, quantized it to under 2 bits per weight, and reports a deployment that fits in 60 MB of disk and runs on about 80 MB of RAM at roughly 400 tokens per second on a laptop CPU. No GPU, no cloud, no accelerator runtime. The work appeared on the Machine Learning subreddit and the core code is on GitHub. It is part of a wave of low-bit quantization work that has moved from theory to practical implementation in the last six months. Two recent research efforts in particular, the quant.cpp project and a paper on Variable Bit-width Quantization, show the same underlying logic: not every token and not every weight carries the same information value, so storing them all at the same precision wastes memory. The Reddit model is the most aggressive small scale demonstration of that principle to date. The implication for builders is straightforward. If you are building applications that currently call out to a hosted API for every inference, the boundary between local and cloud is shifting. A model this small will not match GPT 4, but it can handle classification, summarization, entity extraction, and simple formatting tasks at speeds that are impossible over a network call. The economics change when the model ships inside your application binary instead of living behind a billing endpoint. The compression technique, explained in the post and in the code, has two moving parts. First, the model weights are quantized to under 2 bits each, which means each weight is represented by less than two binary digits, packing more weights into the same number of bytes. Second, the key value cache, the memory the model uses to remember its own previous outputs during a conversation, is handled with a hybrid approach. The most recent 2048 tokens stay in full 16 bit precision, while everything older is compressed. This is the same logic that quant.cpp applies at a different scale. That project, from Quantum AI, demonstrates that keeping the last 128 tokens in FP32 while compressing everything else to 4 bit yields FP32 quality at roughly 3x memory compression. The quant.cpp result was measured on Llama 3.2 3B at 3970 tokens, where the 4 bit progressive approach achieved a perplexity of 19.39 against a baseline of 19.41, a 0.1 percent difference, while reducing KV cache memory from 7.17 GB to 2.33 GB and running 13 percent faster. The Reddit model applies the same insight at a much smaller scale, keeping 2048 tokens at full precision and compressing the rest. The core insight is that attention concentrates on the recent past. quant.cpp reports that roughly 70 percent of attention weight falls on the last 128 tokens, which means the model barely looks at older context. Compressing that older context aggressively costs little in quality because the model was not paying attention to it anyway. The Reddit model extends this to long context by keeping 2048 tokens in fp16 and compressing older tokens, allowing effectively infinite scrollback without the memory growing linearly with conversation length. quant.cpp demonstrates the same effect at larger scale, showing that on a 16GB M1 Pro, a Llama 3.2 3B model with 6.4x KV compression can handle 128K tokens of context in 9.5 GB of RAM, where the FP32 version runs out of memory. Generation speed at 16K context is essentially unchanged, 6.6 tok/s compressed versus 6.5 tok/s FP32. The Variable Bit-width Quantization paper, published on arXiv, pushes the logic further. Instead of assigning a single bit width to every weight, it trains the model so that each group of 64 weights learns its own precision from a set of 1, 2, 4, or 8 bits. The result is a pattern where 69 percent of weight groups collapse to 1 bit, while sensitive parts like the first MLP block keep around 2.5 bits, and the language model head averages 1.09 bits. A 131M parameter model trained this way reaches perplexity 4.2 on TinyStories, beating a 55M FP16 model at 4.4, while using 3.8x less storage. The paper calls this the bigger-but-smaller regime. The recipe is stable enough to freeze into a fixed assignment and reuse without further search, which means the bit allocation pattern can be baked into the deployment. The Reddit model, quant.cpp, and VBQ all share the same architecture for thought: precision is a resource, and it should be spent where it matters. The difference is scale. The Reddit model is a 250M parameter network in 60 MB. quant.cpp applies the same idea to a 3B model and gets a 6.4x reduction in KV cache memory. VBQ scales the concept to 1.46B parameters and shows that the bit allocation recipe transfers across model sizes. And at the far end of the spectrum, the mlx-optiq project shows that 2-bit quantization combined with SSD streaming can fit a 122 billion parameter model on a 36 GB laptop, with the quantized model occupying 44 GB on disk and only 12 GB in RAM during generation. The common thread is that uniform precision is wasteful. Every one of these projects saves memory by spending bits unevenly. If you are building applications that depend on LLM inference, here is what this changes for you. The first cost line that moves is API spend. A 250M model running at 400 tok/s on a laptop CPU can handle tasks that currently route through a hosted endpoint. Classification, summarization, key extraction, and formatting are all within reach of a model this size, especially if it is trained or fine-tuned for the task. If your application does 100,000 inferences a day on a hosted model, and you can move 30 percent of those to a local model, the savings are direct. The second cost line is memory. A 60 MB model that runs in 80 MB of RAM can ship inside your application. That is not a microservice. It is a static asset. The third factor is latency. 400 tok/s on a laptop CPU is faster than any network call. For interactive applications, the difference between 50 ms local inference and 300 ms network round trip is the difference between a feature feeling instant and feeling like a search query. The long context technique matters for agents and chat. The hybrid KV cache approach, keeping recent tokens full precision and compressing the rest, means a local model can maintain long conversations without memory growing linearly. quant.cpp shows this scales to 128K tokens on a 16GB machine. The Reddit model applies the same principle at smaller scale. For agent frameworks that maintain conversation history, this means the memory ceiling is no longer the hard constraint it was with flat FP16 KV caches. This is not a replacement for frontier models. A 250M model trained on 30B tokens will not write code at the level of a 70B model. It will not reason through complex multi-step problems. It will handle structured output generation, simple classification, and pattern matching tasks that currently consume API budget. The strategic question is where to draw the line. The emerging answer, supported by quant.cpp, VBQ, and the Reddit model, is that the line is lower than most builders think. quant.cpp reports its 4-bit progressive approach runs at 19.39 perplexity against a 19.41 baseline, a 0.1 percent quality loss for 3x memory compression. VBQ reports a 131M model at 1.82 mean bits beating a 55M FP16 model. The Reddit model fits in 60 MB. The threshold for good enough has moved. For a builder, the practical move is to audit your inference workload. Break down what your application actually asks of the model. If a significant portion of your calls are structured tasks like classification, formatting, extraction, or simple generation, a local sub-2-bit model may handle them at acceptable quality. The rest, the complex reasoning, the long-form generation, the multi-step planning, can stay on the API. The split is the strategy. The hard part is the training pipeline. The Reddit developer trained from scratch on 30B tokens of FineWeb. VBQ requires training time optimization where the bit allocation is learned jointly with the model. quant.cpp is a post-training method that can be applied to existing models. The effort scales differently depending on which approach you take. Building a local model from scratch requires GPU time for training. Applying quant.cpp to an existing open weight model is a deployment step. Using a VBQ style recipe is somewhere in between, since the recipe can be frozen and reused. The other open question is evaluation. A 250M model that fits in 60 MB is only useful if it does the task well enough. The Reddit post does not include benchmark results. quant.cpp provides perplexity numbers. VBQ provides perplexity and a distributional analysis. mlx-optiq acknowledges that scoring a 2-bit quant of a huge model runs into a wall, noting that at 5 tokens a second through SSD streaming, GSM8K alone is hours. For any local model deployment, the evaluation burden is on the builder. You need to test your specific task against the quantized model, not rely on general benchmarks. The work also raises a question about the model supply chain. If sub-2-bit quantization is practical, the bottleneck shifts from inference to training. The Reddit model was trained on 30B tokens. VBQ requires joint training of bit allocation. The models that benefit most from aggressive quantization are the ones built with it in mind from the start. The frontier model providers are not training with sub-2-bit deployment in mind. The open source community is. That gap will widen. The cloud API model assumes you pay per token. The local model model assumes you pay once for the weights and run them as long as the hardware holds. Sub-2-bit quantization makes the second model viable at scales it was not before. A 60 MB model that runs at 400 tok/s on a laptop CPU is not a curiosity. It is a deployment target. The work that is missing is the evaluation and the ecosystem. The Reddit model has no published benchmarks. quant.cpp has perplexity numbers but no downstream task evaluation. VBQ has perplexity and distributional analysis but acknowledges its method is a from-scratch training phenomenon that does not adapt cleanly to pretrained checkpoints. The next six months will show whether these methods can be combined: whether a VBQ style training recipe can produce a model that quant.cpp style post-training can further compress, and whether the resulting model can match a larger API hosted model on specific tasks. For now, the takeaway is that the floor for local LLM inference has dropped. A model that fits in 60 MB and runs at 400 tok/s on a laptop CPU changes the calculation for any application that currently pays per token for simple inference. The line between local and cloud is not where you think it is. It is lower. ## What is sub-2-bit quantization and why does it change the game for local inference? The core idea is simple. Standard quantization reduces the precision of model weights from 16 bit floating point to 8, 4, or 3 bit. Sub-2-bit quantization goes further, packing weights into less than two bits each. A 250M parameter model at 2 bits per weight needs about 62.5 MB of storage. The Reddit model reports 60 MB, which implies an average of roughly 1.92 bits per weight. The breakthrough is not the number itself. It is that the model still functions. The technique relies on the fact that not all weights contribute equally to output. The VBQ paper demonstrates this formally. When each group of 64 weights is allowed to learn its own precision, 69 percent of groups collapse to 1 bit. The model learns which weights need more precision and which can be compressed aggressively. The result is a heterogeneous bit allocation that is impossible to achieve with uniform quantization. The same principle applies to the KV cache. quant.cpp shows that keeping the last 128 tokens at FP32 while compressing everything else to 4 bit maintains FP32 quality. The Reddit model applies this to the KV cache by keeping the most recent 2048 tokens in fp16 and compressing older tokens. Both methods exploit the same insight: attention concentrates on the recent past, and older tokens can be compressed more aggressively without quality loss. The game changer is the deployment footprint. A 60 MB model can ship inside a mobile app, a CLI tool, or a browser extension. It does not need a server. It does not need a GPU. It does not need a runtime that downloads gigabytes of weights. It is a static asset. ## How does the 60 MB model actually work under the hood? The Reddit model combines weight quantization and KV cache compression. The weights are quantized to under 2 bits each. The KV cache uses a hybrid approach where the most recent 2048 tokens stay in fp16 like a normal KV cache. Everything older gets compressed. This mirrors the quant.cpp approach, which keeps the last 128 tokens in FP32 and compresses the rest to 4 bit. The difference is scale and aggressiveness. The Reddit model is 250M parameters, while quant.cpp demonstrates on Llama 3.2 3B. The Reddit model keeps 2048 tokens at full precision, while quant.cpp keeps 128. Both use the insight that older context can be compressed more aggressively. quant.cpp reports its 4-bit progressive approach achieves 19.39 perplexity against a 19.41 baseline, a 0.1 percent quality loss, while reducing KV cache memory from 7.17 GB to 2.33 GB. The Reddit model does not publish perplexity numbers, but the approach is the same: spend precision where it matters, compress where it does not. The VBQ paper adds a training time dimension. Instead of post-training quantization, VBQ learns the bit allocation during training. Each group of 64 weights learns its own precision from 1, 2, 4, or 8 bits through a Gumbel-Softmax relaxation. The result is a bit allocation that is optimized for the model specific task, not a generic compression ratio. The paper reports that a 131M parameter model at 1.82 mean bits reaches perplexity 4.2 on TinyStories, beating a 55M FP16 model at 4.4. ## What does this mean for your codebase and your API bill? If you are building with LLMs today, your architecture likely has a pattern: user input goes to a hosted API, the API returns a response, you pay per token. Sub-2-bit quantization breaks that pattern. Here is what it means for you. - API spend: A 250M model running at 400 tok/s on a laptop CPU can handle classification, summarization, key extraction, and formatting tasks that currently route through a hosted endpoint. If your application does 100,000 inferences a day on a hosted model, and you can move 30 percent of those to a local model, the savings are direct. - Memory budget: A 60 MB model that runs in 80 MB of RAM can ship inside your application binary. You do not need a separate model server or a large download. The model is a static asset. - Latency profile: 400 tok/s on a laptop CPU is faster than any network call. For interactive applications, the difference between 50 ms local inference and 300 ms network round trip is the shift between a feature feeling instant and feeling like a search query. - Agent memory: The hybrid KV cache approach means a local model can maintain long conversations without memory growing linearly. quant.cpp shows this scales to 128K tokens of context in 9.5 GB of RAM on a 16GB machine. - Deployment surface: You can ship a model to edge devices, mobile apps, or embedded systems without a GPU or constant cloud connectivity. The strategic implication is that you can split your inference workload. The simple tasks stay local. The complex tasks go to the cloud. The split is the strategy. ## Should you train your own or wait for the ecosystem to catch up? The Reddit model was trained from scratch on 30B tokens of FineWeb. That is a real investment. 30B tokens of FineWeb is a substantial training corpus. VBQ requires training time optimization where the bit allocation is learned jointly with the model. quant.cpp is a post-training method that can be applied to existing models. The effort scales differently depending on which approach you take. If you want to replicate the Reddit result, you need a training pipeline. The developer used FineWeb, which is an open dataset. You need GPU time for training and a quantization pipeline. The VBQ paper notes that bit assignments lock in by roughly 20 percent of training, after which they can be frozen into a fixed recipe and reused without further search. That means the expensive search is a one-time cost. If you want to apply quant.cpp to an existing open weight model, the path is simpler. The quant.cpp method is a post-training compression technique. You take an existing model, apply the progressive 4-bit quantization to the KV cache, and keep the last 128 tokens at FP32. The quant.cpp project reports this can be done with a simple API. If you want to use a VBQ style recipe, the path is somewhere in between. The VBQ recipe can be frozen into a fixed bit allocation and reused. But the paper acknowledges that scaling the search beyond 1.5B parameters and adapting it to pretrained checkpoints remain open problems. The practical question is whether the ecosystem will produce tools that make sub-2-bit quantization a deployment step rather than a training pipeline. quant.cpp is already there for KV cache compression. VBQ is a training time method that produces a reusable recipe. The Reddit model is a proof of concept that the full stack works end to end. ## Can you trust a model this small on real tasks? The evaluation question is the hardest one. The Reddit post does not include benchmark results. quant.cpp provides perplexity numbers. VBQ provides perplexity and a distributional analysis. mlx-optiq acknowledges that evaluating a 2-bit quant of a huge model runs into a wall, noting that at 5 tokens a second through SSD streaming, GSM8K alone is hours. For any local model deployment, the evaluation burden is on the builder. You need to test your specific task against the quantized model, not rely on general benchmarks. The VBQ paper provides the most rigorous evaluation. It reports a 131M model at 1.82 mean bits reaching perplexity 4.2 on TinyStories, beating a 55M FP16 model at 4.4. It also reports a 1.46B model on FineWeb-Edu matching a 593M FP16 control at roughly 3.7x less storage. The paper complements perplexity with a distributional analysis using KL divergence and argmax-flip rate, which reveals that deeper layers progressively self-heal quantization error injected by early layers. 
Engineering
A two-bit LLM runs on a laptop in 60 MB and hits 400 tok/s
Sub-2-bit LLM quantization puts a 250M model in 60 MB of disk and 80 MB of RAM, running at 400 tok/s on a laptop CPU. The method keeps recent tokens at full precision and compresses the rest, pointing toward a new class of lightweight deployments.
