Text-to-SQL accuracy: why benchmark scores don't survive your ERP

A text-to-SQL demo is easy to make look perfect. Point a model at a clean eight-table sample database, ask for last quarter's revenue by region, and it writes correct SQL first time. Point the same model at a live NetSuite account with 200 record types, 300 custom fields and four subsidiaries, and text-to-SQL accuracy drops sharply. Nothing about the model changed. The schema did, and so did the question.

This matters because the failure is quiet. Wrong SQL does not usually throw an error. It runs, returns a number, and the number looks reasonable. That is worse than a crash.

What text-to-SQL accuracy actually measures

Published accuracy figures almost always come from one of two measures. Neither is the thing you care about.

Execution accuracy has a blind spot that matters enormously in ERP: a query can execute, return rows, and be wrong. Filter on a status value that exists but means something else and you get a plausible total with no warning. We covered the underlying mechanism in why AI hallucinates on ERP data — the model is not lying, it is completing a pattern, and a pattern that reads well is not the same as a pattern that is true of your account.

Six reasons accuracy collapses on a real ERP schema

Schema size and naming entropy

Benchmark databases have between five and thirty tables with human-readable names. A production ERP has hundreds of record types, and the names are historical rather than logical. In NetSuite, transaction and transactionline hold sales orders, invoices, purchase orders, journals and item receipts in the same structures, separated by a type column. Nothing in the names tells you that. A model that has never seen your account has to infer the mapping from hundreds of candidates, and inference at that width is where errors enter.

Custom fields carry the business logic

The fields that matter most to your business are usually the ones nobody outside your company has ever seen: custentity_credit_tier, custbody_original_po, a custom record holding freight quotes. These are invisible to any model trained on public data. They are also frequently the correct answer. If credit tier lives in a custom field, a query about credit exposure that ignores it is not slightly wrong, it is answering a different question. This is the whole argument for a per-account ERP data dictionary instead of generic schema knowledge.

Business terms have no fixed definition

Ask ten people in one company what revenue means and you get at least three answers: invoiced, recognised, or collected. Open is worse. An open sales order might mean not yet fulfilled, not yet billed, or not yet closed, and those three sets overlap without matching. Customer can mean the billing entity, the parent, or the ship-to. None of this is a SQL problem. It is a definitions problem that SQL then encodes. If you have ever argued about whether DSO should exclude cash sales, you already know the shape of it.

Status domains have to be observed, not assumed

Statuses are the most common source of quietly wrong answers. Status values are shaped by your workflow and your customisations, and the label a user sees rarely equals the value stored on the record. A model asked for overdue invoices will cheerfully write a filter on a status called Overdue, because that is what the phrase suggests. No such status exists. Overdue is a computation over due date, amount remaining and transaction type. The only reliable route is to read the distinct values that actually exist in the account and filter against those.

-- guessed: a status label the schema has never contained
WHERE t.status = 'Overdue'

-- grounded: real ageing logic, with status values read from the account
WHERE t.type = 'CustInvc'
  AND t.status IN ( /* observed open-invoice values for this account */ )
  AND t.duedate < CURRENT_DATE
  AND t.foreignamountunpaid > 0

Several join paths are valid, only one is right

Between a customer and an item there are several defensible routes: through sales orders, through invoices, through fulfilments, or through the item's own sales history. Each returns a different number, and each is correct for a different question. Benchmarks with a single foreign-key path never test this. Real schemas force a choice the model has no basis to make. Ambiguity here is not a model failure, it is a missing requirement.

The question itself has no ground truth

Every benchmark item ships with a reference query, which means a human already decided what the question meant. In production nobody has. “How much are we owed?” has no reference answer until someone rules on credit memos, unapplied cash, intercompany balances and disputed invoices. Any accuracy figure therefore measures agreement with one person's interpretation. Change the interpreter and the score changes, without a single line of SQL changing.

What raises accuracy in practice

None of the fixes are about a bigger model. They are about narrowing the space the model has to guess in, and checking its work before a number reaches a human.

1. 1.Ground the query in the account's real schema. Retrieve the record types, fields, status domains and join paths relevant to this specific question, and put them in front of the model. That converts an open-ended generation problem into a constrained one. It is the difference between writing SQL from memory and writing it with the dictionary open. What grounded ERP AI means covers the mechanism in full.
2. 2.Validate before you execute. Parse the generated query and check every table, column and join against the schema. A field that does not exist is then caught structurally rather than empirically. This removes the most embarrassing failure class entirely, because a hallucinated column cannot survive validation.
3. 3.Execute defensively, then sanity-check the result. Run with row limits and a timeout. Look for the signatures of a wrong query: zero rows where rows are expected, a count larger than the base record population, a total that moved by an order of magnitude against last period. Cheap checks catch a surprising share of silent errors.
4. 4.Ask instead of assuming. When a term is genuinely ambiguous, the correct output is a question, not a query. “Revenue — invoiced or recognised?” takes the user eight seconds to answer and removes an entire error class. A system that never asks is optimising for demo smoothness.
5. 5.Show the query, every time. Verification cannot be delegated to the thing being verified. When the SQL sits under the answer, someone who knows the business can spot a wrong filter in seconds, even without reading SQL fluently. Shown work turns an opaque claim into a checkable one.
6. 6.Refuse when the answer isn't derivable. If the data needed is not in the account, or the request maps to nothing in the dictionary, saying so is the correct output. A system that always answers has, by construction, no way to tell you when it should not have.

How to evaluate text-to-SQL on your own data

You can run a credible evaluation in an afternoon. It needs no framework, only discipline about what counts as correct.

1. 1.Write down 30 questions your team actually asks, in the words they use. Include five you know are ambiguous.
2. 2.Have the person who owns each number define the correct answer in writing, including the definitional choices. This is the expensive step, and the one that makes the exercise worth doing at all.
3. 3.Score three things separately: did it produce a query, did the query execute, did the number match. The gaps between the three are the real finding.
4. 4.Score the ambiguous five differently. The right result there is a clarifying question. A confident answer is a fail, however good the number looks.
5. 5.Repeat monthly. Custom fields get added and statuses get renamed, so a system that is not re-grounded will drift away from your account.

Where ERPray draws the line

ERPray treats accuracy as a property of the whole system rather than the model. During onboarding it builds a data dictionary of your specific account: record types, standard and custom fields, subsidiaries, real join paths, and status domains read from your data instead of assumed. Queries are generated from that dictionary and validated against it before they run, which makes a hallucinated field structurally impossible. Every answer carries the exact SuiteQL that produced it. When a question is ambiguous it asks, and when the answer is not in the account it says so.

That is a narrower promise than a benchmark percentage, and deliberately so. If you want to see how much definitional choice hides inside a single number, the AR aging calculator makes it concrete: bucket boundaries, credit memos and the reserve rate each change the overdue figure without anyone writing bad SQL.