Self-hosted AI for ERP: who needs it, and what it costs you

Self-hosted AI for ERP gets discussed as an ideological position. It is really a procurement constraint. Some organisations cannot send a customer name to an external API, for reasons written into a contract or a regulation rather than a preference. Others can, and choose not to because nobody has asked the question properly. Those two groups should make different decisions, and the second group is much larger than it thinks.

This is the honest version of the trade-off: what running locally buys you, what it costs you, and how to tell which side of the line you are on.

What self-hosted actually means

Four deployment shapes get conflated, and they have very different properties. The question that separates them is not where the application runs, but where the model inference happens.

Most objections people describe as needing self-hosting are actually satisfied by row two or three. It is worth checking which clause you are trying to satisfy before you accept the costs of row four.

Who genuinely needs self-hosted AI for ERP

Five situations where local inference is a requirement rather than a preference.

• Export-controlled technical data. Defence and aerospace supply chains handle drawings, part specifications and programme names where transmission to a third party is restricted by law and by the primes' flow-down clauses. The ERP holds part numbers and customer names that are themselves controlled. This is the clearest case, and it usually comes with an audit obligation attached.
• Explicit contractual bans on external processing. Plenty of enterprise agreements and government frameworks state that no data may be processed by a subprocessor without prior written approval. Adding a model provider is adding a subprocessor. Local inference removes the question rather than answering it.
• Data-residency rules with no compliant region. Where the data must remain in a specific country and no model provider offers inference there, self-hosting is the only route. Note that this is a residency problem, not a security problem, and the fix is geographic rather than architectural.
• On-prem ERPs. If you run Infor SyteLine in your own data centre, the database has never had a path to the internet and your network team is not going to open one for a chat feature. Putting the AI layer next to the ERP is the path of least resistance rather than the paranoid option.
• Air-gapped or classified networks. Manufacturing sites, utilities and defence facilities where the operational network has no external connectivity by design. Nothing except local inference works, and everything about the deployment follows from that.

Who doesn't

If you run a cloud ERP, your ERP vendor already holds every record in question on their infrastructure. In that situation, insisting the AI layer run on your own hardware is not a bad decision, but it is not the security decision it feels like — and the ability to inspect and destroy everything matters more than the postcode of the server.

What most cloud-ERP buyers actually want is four specific guarantees: not sharing infrastructure with other customers, not having their data retained, not having it used to train anything, and being able to see every query that ran. All four are contractual and architectural properties that a dedicated hosted instance can provide. Get them in writing, and you keep frontier-model quality. The AI and ERP security checklist sets out how to ask for each one so the answers are verifiable.

What running locally costs you

This is where vendors go quiet, so plainly: the models you can run on hardware you would actually buy are weaker than frontier models. Not unusable, and not weaker at everything, but the gap is real and it shows up in specific places.

• Long multi-step reasoning. A question that needs five joins, two subqueries and a definitional judgement is where small models degrade fastest. Simple lookups and aggregations are largely fine.
• Robustness to odd phrasing. Frontier models recover from a badly worded question. Smaller models are more literal, so users learn to phrase things carefully — which they do, quickly, but it is friction on day one.
• Context window. Less room to hold a large schema slice, a long conversation and a set of intermediate results at once. This forces sharper retrieval, which is good engineering and extra work.
• Improvement cadence. A hosted model gets better without you doing anything. A local model improves when you upgrade it, which is a change-managed event in a regulated environment.
• Operations. You now own uptime, patching, capacity and backups for one more system. Modest, but not zero, and it lands on a team that is already busy.

What it buys you

• A closed answer to the subprocessor question. No external calls means no data-processing agreement to negotiate, no region to verify, and no annual review of a third party's practices.
• Residency that is a fact rather than a setting. The data is where the server is. That is easy to evidence to an auditor and hard to get wrong.
• Immunity to a vendor's outage or policy change. Your ability to ask questions does not depend on someone else's rate limits or terms.
• Predictable cost. Fixed infrastructure instead of per-token consumption, which matters when usage is broad and dull rather than spiky.
• It works air-gapped. For some sites this is the entire argument, and it is not negotiable.

What fits on modest hardware

More than people expect, because the model is not carrying the whole load. A grounded ERP AI stack spends most of its effort on retrieval, schema validation and query execution, and those are cheap. The model's job is narrower: turn a question into a query using a schema slice it has been handed, and explain the result. Narrow jobs suit small models.

In concrete terms, a complete ERPray instance — the chat stack plus local base models — runs on a single 16 GB server. That is a specification you can satisfy with a virtual machine in your own cloud account or a box in a rack, without a GPU procurement cycle and without a capital request. Expect it to be slower than a hosted frontier model and to prefer clearly phrased questions.

What that hardware will not do is run a frontier-scale model. Anyone telling you a 16 GB box matches the best hosted models is selling something. What it will do is answer the large majority of everyday ERP questions correctly, with the query shown, without a single packet leaving your network.

The air-gapped case

Air-gapped deployment is the strictest version and worth understanding even if it is not you, because it exposes what actually depends on connectivity.

• Nothing can phone home. Not telemetry, not licence checks, not error reporting, not model downloads. Every one of these is a normal part of software that has to be removed or made optional.
• Updates arrive as artefacts. Container images and model weights come in on approved media, get scanned, and get promoted through your own process. Plan for a release cadence measured in months.
• Everything must be inspectable offline. Your security team will want to read what the system does without a vendor portal. That is a reasonable requirement and a good filter on vendors.
• The ERP is usually on-prem too. Which simplifies the architecture considerably: the AI layer and the system of record sit on the same network, and the question of where data travels becomes trivial.

How to decide

Work from the constraint, not the preference. Find the specific clause, regulation or network fact that rules an option out, then pick the least restrictive shape that satisfies it.

Count the internal cost honestly on both sides. Self-hosting adds infrastructure, patching, monitoring and an upgrade process, all of which land on people you already employ, which is exactly the cost that gets left out of comparisons. The ERP TCO calculator has a place to put it, and if you need to justify the whole programme rather than just the deployment shape, the ERP ROI calculator is the sibling worth filling in.

ERPray's position

Every customer runs a dedicated instance. Self-host it on your own cloud or your own hardware, or have it hosted for you on infrastructure that serves only you — never a shared tenancy either way. A complete instance, chat stack plus local base models, fits on a single 16 GB server. Run it fully self-hosted with local models and it makes zero external calls. Your ERP stays the system of record: queries run live against it rather than against a copy, and no data is retained by default.

The grounding layer is the same in every deployment shape. Queries are generated from a data dictionary of your own account, validated against your real schema before they run, and shown to you underneath every answer. That is what makes a small local model workable — see what grounded ERP AI means for the mechanism, and MCP for ERP if you plan to point your own agents at it.