Engineering
LLM performance drift: why your benchmark scores keep moving
LLM performance drift is the silent regression of API-served models over time. Between-day variance hits 8.4 points, three times the within-day noise of 2.8.
#llm
16 stories tagged llm.
Engineering
Hourly LLM benchmark variance: 8.4 points between days
LLM benchmark variance is real and time-dependent. An analysis of 31,352 hourly scores shows 8.4-point swings between days and 2.8 within a day, meaning a single benchmark run can mislead your model choice.
AI
IBM Granite 4.2 ships reasoning and agent RL under Apache 2.0
IBM Granite 4.2 is a family of dense reasoning LLMs in 3B, 8B, and 30B sizes with native chain-of-thought and multi-stage agentic RL, all released under Apache 2.0.
Engineering
A two-bit LLM runs on a laptop in 60 MB and hits 400 tok/s
Sub-2-bit LLM quantization puts a 250M model in 60 MB of disk and 80 MB of RAM, running at 400 tok/s on a laptop CPU. The method keeps recent tokens at full precision and compresses the rest, pointing toward a new class of lightweight deployments.
Research
LLM instruction following collapses past 6 constraints
LLM instruction following breaks past 5 to 6 simultaneous constraints. Per-constraint pass rates decay at 72% times 0.922 per step, probe success below 2% by k=9.
AI
Gemini 3.7 Flash lifts coding scores at half the price
Gemini 3.7 Flash is Google's coding and agent workhorse. It beats 3.6 Flash across every benchmark Google published and undercuts Claude Sonnet 5 on output token price by roughly two-thirds, at $0.75 per million input tokens.
Engineering
Quantized MoE route flips: easy to detect, hard to fix
Quantized MoE route flips silently corrupt expert selection. Detection is easy, but knowing whether to fix each flip requires causal damage analysis.
Research
Knowing-saying gap: probes catch LLM errors confidence misses
Linear probes detect LLM errors that confidence signals miss, exposing a knowing-saying gap. Probe AUROC hits .931 versus .508 for logprob confidence within incorrect trials.
Research
Format Sensitivity Index exposes LLM benchmark gaps
The Format Sensitivity Index measures how LLM benchmark scores shift when prompt wrappers and schema constraints change. The metric exposes a blind spot in model evaluation that developers ignore at their peril.
Research
CreativityNeuro steers LLM weights to break mode collapse
CreativityNeuro is a data-free weight steering method that improves LLM divergent thinking by up to 14 percentile points and reduces mode collapse. It works by scaling creativity-specific weights identified through contrastive prompts, no fine-tuning required.
AI
LLM groupthink now has a novelty benchmark problem
LLM groupthink is the tendency of models to converge on similar answers. Flint scores 7.47 distinct replies out of 10 in Springboards tests.
AI
How to make an LLM escape its own prior with geometry
Novel Search Space breaks an LLM out of its prior by ranking 80,000 dictionary words by embedding distance, banning the obvious neighbours, and forcing the model to brainstorm only from a surprising-but-related band.
Research
LLM recommendation bias gives famous brands an edge
LLM recommendation bias is a measurable incumbent edge: a new arXiv paper found famous skincare brands were recommended 100 percent of the time.
AI
Generative engine optimization: how to rank inside an LLM
Generative engine optimization (GEO) is the practice of getting your content cited inside AI answers from ChatGPT, Perplexity and Google's AI Overviews. Here is what earns a citation and what to change on your site.
Engineering
What is MCP? The Model Context Protocol for data engineers
MCP is an open standard that lets AI models call your tools and data through one connector instead of a custom integration per model. Here is what it means for data engineers.
Snowflake
Snowflake Cortex: running LLMs where your data already lives
Snowflake Cortex AI exposes LLMs as SQL functions like AI_COMPLETE and AI_CLASSIFY, so you run inference without moving data out. It is billed per token by model, and cost swings roughly 40x between a small model and a frontier one, so model choice is the budget.