Every builder who ships an LLM pipeline hits the same wall: the model says it is confident, but is it right? You check log probabilities. You set temperature. You maybe ask the model to verify its own answer. New research on what its authors call the "knowing-saying gap" says all of those signals are leaving a large fraction of errors undetected, because linear probes applied to a model's internal activations can catch mistakes that the model's own confidence metrics miss entirely.
The finding lands at a moment when production LLM systems lean heavily on confidence as a proxy for correctness. Guardrails, routing decisions, human-in-the-loop triggers, and cost controls all assume that when the model is unsure, it says so. Closely related work published earlier this year quantifies the gap with hard numbers: within incorrect trials, logprob-based confidence scores an AUROC of .508, barely above random chance, while a probe-based detection method reaches .931. That is a structural blind spot in the most common error signal in the field.
What did the knowing-saying gap paper actually find?
The paper, posted to arXiv on August 11, 2026, introduces the concept of a "knowing-saying gap" in large language models. The core claim: models often internally represent information about whether their answer is correct, but that information does not surface in their expressed confidence or verbalized uncertainty. Probes, which are simple linear classifiers trained on a model's hidden activations, can extract this signal. Confidence scores cannot.
This builds on a wave of work all pointing the same direction. A study titled "Knowing What You Know Is Not Enough" found that LLM confidences do not align with the model's own actions, meaning a model can internally register that an answer is wrong without acting on that knowledge. Separate work on the detection-extraction gap showed that models can represent correct information internally before they can verbalize it, suggesting internal representations outpace output. And research asking whether LLMs can faithfully express their intrinsic uncertainty in words concluded that verbalized confidence systematically underestimates what models know internally.
The knowing-saying gap paper ties these threads together with a direct comparison: when probes and confidence metrics race on the same errors, probes win by a wide margin. The gap is largest exactly where it matters most, on the trials the model gets wrong.
How much worse is confidence than probes at catching errors?
The quantitative backbone comes from closely related work on how LLMs detect and correct their own errors. In a study of 7,223 trials across multiple models, researchers compared six methods for predicting whether a model's answer is correct:
- Logprob: the model's raw log-probability for its answer, scoring .508 AUROC within incorrect trials
- Confidence: a verbalized or computed confidence score, at .657
- Verification log-probability difference: .907
- LD, a detection-based metric: .901
- Combined behavioral model: .921
- PANL, a probe-based detection method: .931
The chart below shows the gap plainly.
Logprob confidence, the signal most production systems rely on, barely beats random guessing at .508 AUROC. The probe-based method reaches .931, a gap of .423 AUROC points. Confidence at .657 is better than logprob but still leaves a large detection blind spot.
The same study found that PANL adds significant predictive variance beyond the combined behavioral baseline, with a likelihood ratio chi-square of 237.7 at p < .001. Within the behavioral subset of 856 trials where the model actually changed its answer, probe-based detection retained predictive value at p < .01, while other behavioral features dropped to chance. Probes catch what behavior misses.
Why does this gap matter for my LLM pipeline?
If you build or operate any system that uses LLM confidence to make decisions, this gap is your problem. Here is where it bites:
- Guardrail gating. Most safety and quality guardrails trigger on low-confidence outputs. If confidence misses errors at .508 AUROC, your guardrail is letting roughly half of all wrong answers through unchecked. A probe-based gate would catch the vast majority.
- Human-in-the-loop triggers. You probably route low-confidence outputs to human review. The knowing-saying gap means the model's most dangerous errors, the ones it makes with false confidence, never trigger review. You are reviewing the wrong cases.
- Cost optimization. Temperature, sampling, and chain-of-thought verification all assume the model can assess its own uncertainty. If that assumption is broken, you are spending compute on verification that does not verify.
- Evaluation pipelines. If your eval suite uses confidence as a quality signal, your eval is measuring the wrong thing. A model can be confidently wrong and your eval will not flag it.
- Routing in multi-model systems. When you route queries between models based on confidence, the router is working with a corrupted signal. Strong queries get sent to weak models and vice versa. For more on how this plays out in production, see our coverage of the enterprise AI agent evaluation gap, where half of enterprises ship agents they know are broken.
The business consequence is direct: every system that gates on confidence is shipping with a known, measurable false-negative rate on error detection. The research says that rate is close to 50 percent for the most basic signal.
Should I replace confidence thresholds with probes today?
Not yet, but you should start building toward it. Here is the honest read.
Probes require access to model internals. If you are calling an API-only model from a frontier provider, you cannot extract hidden activations. You are stuck with confidence signals and the gap that comes with them. This is one more reason open-weight models matter for production safety: you can probe them.
If you run open-weight models or have API access that exposes activations, the path is clearer. Train a linear probe on a labeled set of correct and incorrect outputs, using the model's hidden states as input. The probe becomes a lightweight error detector that runs alongside the model. The research suggests this adds substantial detection power beyond any behavioral signal.
There are caveats. Probes are model-specific. A probe trained on Llama 3 will not work on Qwen. Probes may also be distribution-specific: a probe trained on math errors may not generalize to code errors. And probes add engineering complexity that confidence thresholds do not.
The practical bet for a builder today:
- If you run API-only frontier models, accept the gap and compensate with external verification, structured outputs, and ensemble checks. Do not trust self-reported confidence for high-stakes decisions.
- If you run open-weight models, start experimenting with probes now. The AUROC difference is large enough that even a mediocre probe will outperform confidence.
- If you are building evaluation infrastructure, add probe-based error detection as a signal alongside confidence. Compare them. The gap will tell you how much your current system is missing.
- If you are designing multi-agent routing, do not route on confidence alone. Use task complexity, query type, and external validation. The knowing-saying gap means your router is working with partial information.
What comes next for probe-based error detection?
The research is moving fast. Work on closing the confidence-faithfulness gap is trying to make verbalized confidence more honest, which would narrow the knowing-saying gap without requiring probes. If that succeeds, confidence becomes a better signal and probes become less necessary. If it fails, probes become the default error detection layer for any serious deployment.
The open questions are about generalization. Does a probe trained on one task type transfer to others? Does a probe trained on one model family transfer to fine-tuned variants? Does probe accuracy degrade under distribution shift? These are empirical questions that will determine whether probes become infrastructure or remain a research tool.
Watch for frontier providers exposing activation-level APIs. If OpenAI or Anthropic offers a confidence probe endpoint that returns probe-based error scores, the gap closes for API users overnight. Until then, the advantage sits with teams running their own weights.
The gap that guardrails cannot cross
The knowing-saying gap is a structural property of how LLMs work: internal representations carry more information about correctness than output-level confidence can express. Every guardrail, router, and human-review trigger built on confidence is operating with a known blind spot. The research now puts a number on it: roughly half of all errors pass through undetected. The question for builders is whether they can afford to ship without trying.
Sources
- arXiv - The Knowing-Saying Gap: When Probes See Errors that Confidence Misses
- arXiv - How LLMs Detect and Correct Their Own Errors: The Role of Internal Confidence Signals
- arXiv - Knowing What You Know Is Not Enough: Large Language Model Confidences Don't Align With Their Actions
- arXiv - The Detection-Extraction Gap: Models Know the Answer Before They Can Say It
- arXiv - Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?
- arXiv - Closing the Confidence-Faithfulness Gap in Large Language Models
