Dataset: ESQ-Bench public release: schemas, seed scripts, 550 questions, evaluation harness
Your text-to-SQL pipeline reports 89% accuracy. It is probably lying. ESQ-Bench, an Oracle-first NL2SQL benchmark released this week on arXiv, finds that the gap between "query executed successfully" and "query returned the right answer" is catastrophic in enterprise environments. GPT-4o, a model you likely use in production, drops from 79.8% execution accuracy on simple schemas to 57.2% on complex ones. And among queries that pass execution checks, 92% diverge semantically from the gold query at Tier 1 alone.
The headline number: 92% silent divergence among execution-passing queries at Tier 1. If your evaluation harness checks whether generated SQL runs and returns the same rows as the expected query, you are missing nearly all the failures that matter.
The benchmark, built by independent researchers Sanjay Mishra and Divya Chukkapalli with Ganesh R. Naik of Torrens University Australia, targets a blind spot in the NL2SQL field. Every major benchmark, from Spider to BIRD, runs on SQLite or PostgreSQL with academic schemas. ESQ-Bench runs Oracle-first across 465 tables and 164,682 rows, with identical seed data replicated on PostgreSQL, MySQL, and SQL Server. It released 550 gold-validated question-query pairs across three complexity tiers.
What exactly does ESQ-Bench measure?
Four metrics, each catching a different failure mode:
- Exact Match (EM): Does the generated SQL match the gold query text exactly? GPT-4o never clears 7% on any tier. Almost no generated query is textually identical to the reference.
- Execution Match (EX): Does the generated query execute and return the same rows as the gold query? This is the field's dominant metric. GPT-4o scores 79.8% on Tier 1, 60.3% on Tier 2, 57.2% on Tier 3.
- Semantic Recall (SR): Does the query return the correct semantic result, validated against annotated intent? This catches queries that return the same rows for the wrong reasons.
- Silent Divergence (SD): The new metric. A query exhibits silent divergence if it passes EX but fails EM or SR. In other words, it ran, it returned rows, but the SQL is not semantically equivalent to the gold query.
The SD metric is the contribution that should make you rethink your evaluation pipeline. The operational definition flags a query as silently divergent if EX passes but EM or SR fails. At Tier 1, 69 of 75 EX-passing queries are silently divergent, a 92% rate. At Tier 2, it is 55.2% among executed queries. At Tier 3, 56.7%.
This means your text-to-SQL system can report 80% execution accuracy while the underlying SQL is semantically wrong in nearly every case that passes.
How bad is the Oracle dialect problem for NL2SQL?
Oracle Database has syntax and semantics that diverge from SQLite and PostgreSQL in ways that cause silently wrong or explicitly failing queries. The paper enumerates several: FETCH FIRST pagination instead of LIMIT, CONNECT BY hierarchical traversal, MINUS instead of EXCEPT, empty-string-as-NULL coercion, and NULLS LAST default ordering.
If your model trained on SQLite and PostgreSQL benchmarks, it does not know these constructs. When it generates Oracle SQL, it either fails outright or produces a query that runs but returns different rows because Oracle handled NULLs differently than the model assumed.
The benchmark is Oracle-first by design. It is not a dialect translation of Spider or BIRD. Every schema, question, and gold query was built ground-up for Oracle semantics, syntax, and enterprise conventions. The researchers then replicated the identical seed data on PostgreSQL, MySQL, and SQL Server to enable cross-dialect comparison.
The scale is enterprise-grade. The six schemas contain 465 tables and 164,682 rows with zero empty tables. The 550 questions are split across three tiers: 95 in Tier 1, 228 in Tier 2, 227 in Tier 3. Each tier escalates schema complexity with NULL traps, status-domain separation, CONNECT BY hierarchies, entity-overlap joins, and ambiguity-resolution traps.
Other dialect benchmarks exist. UniQL aligns 1,534 natural language questions across 16 SQL dialects, yielding 24,544 dialect-specific queries. BEAVER covers 9,128 question-SQL pairs from real-world query logs across 812 tables. But ESQ-Bench is the first to combine Oracle-first design with the silent divergence metric.
Which models hold up, and which collapse?
The researchers ran two frontier closed models and one open-weight baseline.
| Model | Tier 1 EX | Tier 2 EX | Tier 3 EX |
|---|---|---|---|
| GPT-4o (schema-linked) | 79.8% | 60.3% | 57.2% |
| Claude Sonnet 4.6 (schema-linked) | 87.4% | 74.9% | 68.7% |
| Llama 3.2 (schema-linked) | 13.3% (bank-wide) | , | , |
The chart above shows the execution accuracy degradation for GPT-4o and Claude Sonnet 4.6 across the three complexity tiers. Claude Sonnet 4.6 exceeds GPT-4o on every tier, reaching 87.4% at Tier 1 and 68.7% at Tier 3.
The open-weight baseline tells a stark story. Local Llama 3.2 with schema-linked prompting reaches only 13.3% bank-wide execution accuracy (73 out of 550 questions). The gap between closed API models and open-weight baselines on enterprise Oracle schemas is enormous.
A surprising finding: GPT-4o zero-shot execution match inverts schema-linked at Tiers 2 and 3. Zero-shot scores 78.7% / 73.5% / 77.8% on executed queries. The researchers attribute this to lower execution rates and survivor bias in the zero-shot analysis. When fewer queries execute, the ones that do are disproportionately simpler, inflating the success rate.
This means your prompting strategy matters. Schema-linked prompting, the approach most production systems use, produces lower execution accuracy than zero-shot at higher tiers because it attempts more complex queries. The queries it generates that do execute are harder, and more of them fail.
Why does silent divergence matter for your codebase?
If you are building a natural language interface to an enterprise database, your evaluation harness almost certainly uses execution match. You generate SQL, run it, compare rows to the gold query's rows, and report a percentage. ESQ-Bench shows that percentage is a mirage.
At Tier 1, the simplest tier, 92% of EX-passing queries are silently divergent. The query ran. It returned rows. Those rows matched the gold query's rows. But the SQL itself is not semantically equivalent. On a different dataset, or with different NULL distributions, or at a boundary condition, it would return different results.
The failure analysis backs this up. The paper introduces a four-category taxonomy (F1 through F4). F3 wrong-result semantics dominate: 170 out of 243 classified failures. F1 invalid-identifier errors rise from 1 at Tier 1 to 42 across Tiers 2 and 3 combined, reflecting the complexity of enterprise schema naming conventions.
What this means for you:
- Your accuracy numbers are inflated. If you report 80% execution accuracy on your text-to-SQL pipeline, the real semantically-correct rate is likely under 10% if it follows the ESQ-Bench pattern. Build semantic recall checks, not just row comparison.
- Oracle is a blind spot. If your production database runs Oracle and you evaluated on SQLite or PostgreSQL benchmarks, you have an unmeasured risk. Oracle-specific NULL handling, pagination syntax, and hierarchical queries are failure modes your benchmark did not test.
- Open-weight models are not ready for enterprise Oracle. Llama 3.2 at 13.3% bank-wide is unusable for production text-to-SQL on Oracle. Budget for frontier closed models or invest in fine-tuning with Oracle-specific training data.
- Schema-linked prompting has a survivor-bias trap. Zero-shot can appear to outperform schema-linked at higher tiers because it generates simpler queries and fails to execute on harder ones. Measure execution rate alongside execution accuracy.
- Exact match is nearly worthless as a metric. GPT-4o never exceeds 7% EM. If you use EM as a quality gate, you will block nearly every query your model generates, including the good ones.
What should you do about it?
First, audit your evaluation harness. If it only checks execution match, you are flying blind on semantic correctness. Add a semantic recall layer. The ESQ-Bench paper releases its evaluation harness (run_esq_tier1_evaluation.py) and failure analysis tool (analyze_esq_failures.py) as part of its public release.
Second, if you run Oracle in production, pressure-test your pipeline against Oracle-specific constructs. The paper's tiered complexity design gives you a blueprint: create schemas with NULL traps, CONNECT BY hierarchies, and entity-overlap joins. Run your model against them and measure where it breaks.
Third, if you are choosing a model for enterprise NL2SQL, the data points to Claude Sonnet 4.6 over GPT-4o for Oracle workloads. The gap is 7.6 percentage points at Tier 1 and grows to 11.5 points at Tier 3. But both models still show high silent divergence rates, so model selection alone does not solve the problem.
Fourth, if you are building a cross-dialect system, look at the emerging dialect-specific approaches. Dial, a knowledge-grounded dialect-specific NL2SQL system, reports 10.25% improvement in translation accuracy over baselines. A semantic-layer-mediated agent driven by Gemini 3 Pro reaches 94.15% execution accuracy on the Spider2-snow benchmark, though that benchmark does not measure silent divergence.
Fifth, watch for the multi-model study. The ESQ-Bench researchers plan a full multi-model comparison (Table 11 protocol, Section 6.6). That will give you a broader landscape of which models handle enterprise Oracle schemas best.
The metric you are not measuring
Execution match is a proxy. It is a convenient proxy that lets you run a script and get a number. But it is a proxy for a proxy: the query ran, it returned rows, those rows happened to match. The thing you actually want, semantic correctness, is harder to measure and nobody was measuring it at scale on Oracle until now.
ESQ-Bench does not solve the silent divergence problem. It exposes it. The 92% silent divergence rate at Tier 1 means the field's primary evaluation metric misses nearly every failure that matters. Your text-to-SQL pipeline is probably better than it looks on paper, and far worse than it looks in production.
Sources
- arxiv.org , ESQ-Bench: A Multi-Tier Enterprise Oracle Benchmark for Evaluating NL2SQL Dialect Generalization and Silent Semantic Divergence
- arxiv.org , UniQL: Towards Dialect-Universal Benchmarking for Text-to-SQL
- alphaxiv.org , BEAVER: An Enterprise Benchmark for Text-to-SQL
- arxiv.org , Dial: A Knowledge-Grounded Dialect-Specific NL2SQL System
- arxiv.org , A Semantic-Layer-Mediated Agent for Natural Language to SQL over Heterogeneous Enterprise Databases
