Tag

#gguf

1 story tagged gguf.

Bar chart comparing generation throughput in tokens per second between transformers with torch.compile and llama.cpp for two GGUF checkpoints on M3 Max. Qwen1.5-MoE A2.7B Q4_K_M shows transformers at 103.6 tok/s versus llama.cpp at 78.4 tok/s, a 1.32x speedup. Llama-3.2-3B Q4_K_M shows transformers at 68.8 tok/s versus llama.cpp at 50.2 tok/s, a 1.37x speedup.

Transformers GGUF inference nears llama.cpp speed on Mac

GGUF inference in transformers now matches llama.cpp on Apple Silicon, hitting 103.6 tok/s with ggml Metal kernels while keeping you in PyTorch.

Lars Cornelissen · Sep 22, 2026