Why AI chatbots hallucinate on ERP data

A controller connects a general-purpose AI assistant to their ERP, asks how much AR is overdue, and gets back $3,184,220. It looks right. It is formatted like a real answer. It is wrong — the real figure is $2,640,180, because the query counted credit memos as positive balances and silently excluded one subsidiary. Nothing in the answer hinted at either problem. That is a hallucination, and understanding why AI hallucinates on ERP data matters — because it is a predictable outcome rather than bad luck.

What a hallucination actually is

Hallucination

An output that is fluent, well-formed and confident, but not derived from any verified source. The model is not lying and it is not malfunctioning. It is producing the most plausible continuation of your prompt — and plausibility is a different test from truth.

A language model is a very good next-token predictor trained on an enormous amount of public text. That text includes millions of SQL queries, a great deal of ERP documentation, and thousands of forum threads about NetSuite. What it does not include is your account. Your custom fields, your renamed statuses, your subsidiary structure and your local conventions were never in the training data and cannot be inferred from a question.

So when you ask for overdue AR, the model does exactly what it was built to do: it writes the most probable-looking query for that request. Probable-looking means field names that appear in documentation for some system, joins that would be sensible in a schema, and status literals that match how a human would say it out loud.

-- What an unconstrained model writes for "how much AR is overdue?"
SELECT SUM(i.amount_due) AS overdue_ar
FROM invoice i
JOIN customer c ON c.id = i.customer_id
WHERE i.status = 'Open'
  AND i.due_date < CURRENT_DATE

In a system with a forgiving schema, that query errors out and you know something went wrong. The dangerous case is the near-miss: the query runs, the column names happen to exist, and the number that comes back is arithmetically valid and semantically meaningless. There is no error to catch, only a number that is quietly the wrong number.

Why AI hallucinates on ERP data specifically

Free-text tasks tolerate approximation. If a model summarises a document and gets the emphasis wrong, you notice. ERP questions are the opposite: the output is a single number with no visible provenance, the schema is enormous, and almost every meaningful part of it varies between accounts running the same software.

The word "open" means six different things

Ask an ERP admin what "open" means and you will get a follow-up question. An open sales order might be unapproved, approved but unfulfilled, partially fulfilled, or fulfilled but unbilled. An open invoice has a balance remaining. An open purchase order is unreceived, or received but unbilled. An open period is one that has not been closed. An open case is unresolved. All of these live in the same account and several use the same word in the same status column.

A model picks one interpretation, applies it silently, and does not tell you which. Two people asking the same question in different words get two different numbers, and both look authoritative. If all you actually need is the arithmetic on figures you already trust, an AR aging calculator gives you buckets and weighted days overdue without any of this ambiguity.

Why a wrong number costs more than no number

The instinctive reaction to an occasional bad answer is that it is a fair trade for speed. It is not, because the two failure modes are not symmetrical. No number stops a decision and creates work. A wrong number lets the decision proceed and creates a liability you do not know you are carrying.

• It gets used. A number produced in eight seconds during a meeting goes straight into the decision. Nobody schedules a verification pass on something that arrived that easily.
• It propagates, then surfaces late. The figure lands in a board pack or a covenant calculation, and the error appears at audit or in a customer dispute weeks later. Correcting it means correcting everything downstream.
• It is unfalsifiable after the fact. With no query shown, you cannot reconstruct what was computed, or tell whether the number was wrong or the question was misread.
• It burns the tool. One publicly wrong number ends adoption for a year. Finance teams extend trust slowly and withdraw it immediately, which is the correct instinct.

That said, the slow path has a real price too, and it is worth quantifying before you decide how much verification effort is reasonable. Four people spending six hours a week each assembling reports is a meaningful annual number — the reporting time savings calculator puts a figure on the hours you are currently spending to be safe.

Generic models are not defective. They are unconstrained.

It is worth being fair here, because the loudest version of this argument is unfair. A general-purpose model is not badly built. It is doing sophisticated work with the information it was given, which in the ERP case is almost none. Given a real schema and real constraints, the same model class is very capable at this task. The gap is not intelligence. It is context and enforcement.

What actually fixes hallucination on ERP data

Prompt engineering does not fix this. Neither does a larger model, because the missing information is not in any model's weights — it is in your account. Four structural controls do the work, and together they are what grounded ERP AI means as a term.

1. Ground the query in a per-account dictionary

Before answering anything, read the account: record types, every field including custom ones, the real join paths, the status values that actually occur, row counts. Generate queries from that inventory rather than from memory. A field that is not in the dictionary cannot appear in a query, which is a different guarantee from a model being told not to invent fields. What a per-account data dictionary contains is worth understanding in detail, because everything else depends on it.

2. Validate against the real schema before execution

Generate the query, then check it against the account's schema before it runs: do these tables exist, do these columns exist on these tables, is this join path real, is this status literal in the observed domain. A query that fails validation is rewritten or refused. It never reaches your ERP and never produces a number. This is the step that converts a class of silent wrong answers into visible refusals.

3. Show the query with every answer

Every answer should carry the exact query that produced it, in full, re-runnable. It lets a competent person check the definition instead of trusting the system — usually in fifteen seconds, because the argument is almost always about a WHERE clause. It also changes the incentive structure of the product: work that has to be shown gets built more carefully. If a query arrives as one long line, the SuiteQL formatter makes it readable before you review it.

4. Refuse outside what is known

The most important behaviour is the least impressive one in a demo. If the question needs a field that does not exist, a definition the system has not been given, or data the connected role cannot see, the correct output is a refusal with a reason — not a best effort. "I can't answer that because there is no field recording delivery promise dates" is a useful answer. A number invented to fill the silence is not.

Four questions that expose an ungrounded answer

You can test any AI-over-ERP tool in about ten minutes, including in a vendor demo, using their account or yours.

1. 1.Ask for the query, then read the WHERE clause. If the system shows a paraphrase of its "reasoning" instead, nothing it tells you is checkable.
2. 2.Ask the same question two ways. "Overdue AR" and "how much are customers late paying us" should return the same number. If they differ, the definition is being re-guessed each time.
3. 3.Ask about a custom field by its label. A grounded system uses it correctly or says it cannot find a field by that name. An ungrounded one produces something adjacent and plausible.
4. 4.Ask something the ERP cannot answer — a field that does not exist, or data outside the connected role. The correct response is a refusal.

The same four checks belong in a formal evaluation alongside the access, retention and audit questions in the AI and ERP security checklist. Accuracy and governance fail together: a system that cannot show its query also cannot give your auditors a trail.

Where ERPray draws the line, including the limits

ERPray builds a data dictionary of your specific account during onboarding, generates queries from it, validates them against your real schema before running them, and shows the SuiteQL under every answer. It connects through a least-privilege role you create, and stays read-only until you explicitly enable writes per record type and field. Queries run live against your ERP rather than a copy.

The honest limits: grounding makes invented fields structurally impossible, but it does not make intent unambiguous. If you ask for "revenue" and your team has three definitions of it, the right behaviour is to ask which one — and a shown query is how you catch it when the wrong one was assumed. Grounding also cannot repair the underlying data. If your due dates are wrong in the ERP, every correctly-computed answer about lateness will also be wrong, and no AI layer changes that.