GGUF inference just became a first-class citizen in the Hugging Face transformers library. For the first time, you can load a GGUF file the same way you load any other checkpoint, keep the weights packed in their native quantization, and run them through the same Metal kernels that llama.cpp uses, all without leaving Python. The initial benchmarks on Apple Silicon show transformers matching or slightly exceeding llama.cpp on generation throughput, with a 3.3x memory reduction over bfloat16 for a mixture-of-experts model.
GGUF inference is the practice of running models stored in the GGUF file format, which packages weights, tokenizer, and chat template in one file with configurable quantization levels like Q4_K_M. Tools such as Ollama, LM Studio, and Jan already rely on it, and publishers like Unsloth, bartowski, and the LM Studio Community have made GGUF checkpoints downloadable millions of times on the Hugging Face Hub. Hugging Face announced the integration on September 22, 2026, with the goal of letting developers run those same checkpoints inside transformers without first dequantizing the weights and losing the memory savings.
What did Hugging Face actually ship?
The core change is a new loading path that keeps GGUF weights in their packed format and dispatches computation to ggml's Metal kernels. You enable it by passing gguf_file to from_pretrained:
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained(
"unsloth/Qwen3.5-4B-GGUF",
gguf_file="Qwen3.5-4B-Q4_K_M.gguf"
)When the weights stay packed on Metal, transformers automatically loads compatible ggml Metal layer kernels and uses ggml-org/ggml-attn as the attention implementation. If that kernel cannot be fetched, the model falls back to PyTorch's SDPA attention with a warning. That is the only GGUF-specific step. Everything after it is the standard transformers API: the same generate call, the same tokenizer, the same logits processors.
The supporting pull request, authored by Arthur Zucker and opened May 14, 2026, is substantial at 3,317 lines added across 18 files. It packages the ggml Metal kernels as ArthurZ/gguf-kernels and supports the common quantization types: Q4_0, Q5_0, Q5_1, Q8_0, Q4_K, Q5_K, Q6_K, IQ4_NL, and IQ4_XS. The blog post recommends starting with Q4_K_M, which mixes 4-bit weights with higher-precision tensors for sensitive layers, then trying Q5_K_M or Q6_K if you have more memory available.
Beyond the loader, the PR also fixes several performance bottlenecks. A custom paged-decode attention Metal kernel runs an online softmax reading K/V through block-table indirection, replacing a gather-then-SDPA fallback that took roughly 16 ms per layer with a kernel that takes 134 microseconds per call. A paged KV write kernel collapses a five-operation Python chain into a single dispatch at 6.7 microseconds. And two changes to the generate loop benefit all transformers models, not just GGUF: dropping an unnecessary attention mask early when input has no padding, and deferring the stopping check so the CPU can keep scheduling GPU work asynchronously.
How fast is transformers compared to llama.cpp?
The blog post benchmarks three checkpoints on a MacBook Pro M2 Max with 32 GB unified memory, running macOS 26.6, PyTorch 2.12.1, and kernels 0.17.0. The llama.cpp numbers come from llama-bench reporting tg128, which is decode-only throughput over 128 tokens. The transformers numbers come from generate producing the same 128 tokens from a 12-token prompt, best of three warmed runs, and they include prefill. Hugging Face says transformers is "close to llama.cpp across all three checkpoints" and notes the methodology gap: transformers includes prefill while llama-bench reports decode-only, so the comparison slightly understates transformers' decode speed.
The PR benchmarks on an M3 Max tell a more detailed story. With torch.compile enabled and SDPA attention, the Qwen1.5-MoE-A2.7B model at Q4_K_M hits 103.6 tok/s, which is 1.32x faster than llama.cpp at 78.4 tok/s. The dense Llama-3.2-3B at Q4_K_M reaches 68.8 tok/s, 1.37x faster than llama.cpp at 50.2 tok/s.
The chart above shows the throughput comparison from the PR. Both compiled transformers configurations exceed llama.cpp, though the PR notes that eager execution without compile lands closer to parity: 46.7 tok/s for the MoE model and 49.2 tok/s for the dense model. For batched inference, the results are more mixed. A batch of 8 with Metal flash attention and eager execution hits 103.2 tok/s aggregate for the MoE model, but single-request batched generation with compile drops to 20.3 tok/s, suggesting the compiled path needs work for batched scenarios.
The memory story is the other half. Qwen1.5-MoE-A2.7B at Q4_K_M on an M3 Max uses 8.8 GB of resident weights with packed inference, compared to 28.6 GB in bfloat16, a 3.3x reduction. That is the difference between fitting a model on a 16 GB Mac and needing 32 GB or more. The same packed-path speedup comes from native Q4_K matvec and matmul Metal kernels, which are bandwidth-bound and win 1.27x over bfloat16 matvec for decode.
Why would I run GGUF in transformers instead of llama.cpp?
Hugging Face is explicit in the blog post: llama.cpp remains their recommended engine when your priority is efficient local inference. Its dedicated runtime, memory management, and broad hardware support are built around that goal. So why would you use transformers instead?
The answer is developer ergonomics. You get the performance of native quantized inference without leaving the PyTorch ecosystem. That matters for several workflows:
- Prototyping custom architectures. If you need to inspect intermediate activations with hooks, modify a model's forward pass, or add custom layers, you need Python and PyTorch. llama.cpp is a C++ inference engine, not a research framework. With this integration, you can experiment with GGUF models using the same tools you already use for development.
- Evaluating quantized checkpoints. If you have existing transformers evaluation pipelines, you can now run them directly on GGUF files instead of writing a separate harness that calls llama.cpp. This matters if you want to measure quality degradation from quantization on your specific tasks.
- Validating GGUF conversions. When you convert a model to GGUF, you want to verify the weights were converted correctly. Loading both the original checkpoint and its GGUF conversion in transformers lets you compare them directly, accounting for quantization error.
- Fine-tuning from a GGUF checkpoint. You can dequantize the weights with
GgufConfig(dequantize=True)and continue with a standard transformers training workflow. This is useful if you want to start from a community quantized checkpoint and adapt it. - Trying new decoding strategies. Custom logits processors, stopping criteria, and entirely custom generation loops are all available through the transformers API.
For a team building a local AI application, the practical implication is that you can keep one codebase for both development and local deployment. You prototype in transformers, validate quality, and if you need maximum efficiency in production, you can switch to llama.cpp for serving. The same GGUF checkpoint works in both. The generation loop improvements that reduce CPU-to-GPU synchronization are part of a broader pattern of making inference fast without requiring a separate runtime.
What are the catches?
This is a first release with clear boundaries. The packed inference path is MPS-only, meaning Apple Silicon Macs. On other platforms, transformers can import GGUF files but falls back to dequantization, which uses more memory and gives no speed benefit over loading the original weights.
Architecture coverage is limited to Qwen3.5 dense and MoE models, plus compatible Qwen3.8 checkpoints. If you want to run a Llama or Mistral GGUF in packed mode, you are out of luck for now. Hugging Face says adding support for other architectures is straightforward and they will expand coverage gradually, but the timeline is unspecified.
Padding and batching still need work. The attention mask optimization only applies to unpadded inputs. Padded batches cannot take the same shortcut and can have lower performance. The blog post says they want to extend the work to generate_batch on MPS, but that is future work, not current capability.
The torch.compile results, while impressive, come with the usual caveats. Compilation takes time on the first run, and recompilation can trigger when input shapes change. The blog post emphasizes that their focus was making eager execution fast without requiring compile, because interactive use needs quick start and steady token streams without compilation pauses. The compile numbers are the ceiling; the eager numbers are the floor you get without any configuration.
Finally, the benchmark comparison has a methodology gap worth noting. The transformers numbers include prefill overhead while llama-bench reports decode-only throughput. The PR's M3 Max comparison also uses a different token count (256) than the blog's official benchmarks (128). The numbers are useful for understanding relative performance, but they are not a controlled head-to-head.
What should builders do with this now?
If you are already running local inference on Apple Silicon with llama.cpp and it works for you, there is no urgent reason to switch. This integration is not trying to replace llama.cpp for production serving. If you are building in the transformers ecosystem and want to run quantized models locally on Mac, this is now the path of least resistance. A few concrete steps:
- Pull the latest transformers main branch and install the compatible kernels version. The blog post specifies kernels 0.17.0 and PyTorch 2.12.1. This has not shipped in a stable release yet.
- Start with Q4_K_M quantization for any model you test. It is the recommended default and gives the best speed-to-quality tradeoff for most use cases.
- Evaluate on your actual tasks. The blog post explicitly warns that quality tradeoffs from more aggressive quantization depend on the model and the task. Run your own evaluation, not just a benchmark suite.
- Use the OpenAI-compatible server if you want to connect a chat client.
transformers servewith a<model_id>:<filename>.ggufargument exposes an endpoint that works with Jan, Pi, and any other OpenAI-compatible client. - Watch for architecture expansion. If you need a model that is not yet supported, open an issue on the transformers repo with the checkpoint and your use case. Hugging Face is prioritizing based on developer demand.
For teams deciding between llama.cpp and transformers for local deployment, the decision now hinges on what you need beyond inference. If you need a lightweight, cross-platform server, use llama.cpp. If you need to prototype, evaluate, fine-tune, or inspect model internals in Python, transformers with GGUF support gives you that without giving up much performance.
The real play is kernels, not GGUF
The GGUF integration is the headline, but the deeper strategy is the kernels library. A kernel operates on tensors. It does not require the whole model to come from a GGUF file. The same ggml Metal kernels for attention, normalization, and matrix multiplication can be integrated into any transformers model that uses compatible operations.
That matters because llama.cpp requires a full C++ implementation of every architecture it supports. New research models, custom variants, and architectures that llama.cpp has not yet implemented are out of luck. But transformers already has PyTorch implementations of most architectures. If the ggml kernels can accelerate individual operations within those implementations, you get the performance benefit without waiting for someone to port the entire model to llama.cpp.
This also extends beyond text. Computer vision models, audio models, and multimodal models could reuse compatible attention and matmul kernels without first having a full llama.cpp implementation. Each architecture still needs integration and validation, but the path is open. Hugging Face, which Nvidia agreed to acquire for $12.9 billion, is positioning this as a convergence play: the same quantized checkpoints and the same underlying kernels, accessible from both the C++ runtime optimized for serving and the Python framework optimized for development. For builders, that means the gap between prototyping and production local inference is narrowing, and the choice of tool is becoming less about performance and more about workflow fit.
Sources
- Hugging Face Blog - Transformers now runs llama.cpp quants
- GitHub - GgufLinear: inference-time GGUF matmul on Apple Silicon, PR #45977
- Hugging Face Docs - Transformers optimization overview
- Hugging Face Docs - LLM inference optimization
