Every company hitting scale with AI hits the same two walls: the monthly bill, and the day your provider has an outage. We solved both at once with an in-house LLM gateway: a single layer that every AI request goes through, which decides which model should answer it.

Behind it sat three kinds of model. The expensive, most capable ones from Anthropic and OpenAI, which the industry calls frontier models. Smaller, cheaper ones from the same providers. And open-weight models we ran on our own infrastructure, meaning models whose weights are published so anyone can host them. Routing work to the right one cut AI running costs by up to 80% with no meaningful drop in output quality. And when both our first and second choice providers had outages lasting hours, 700+ employees kept working without noticing.

80%
Peak reduction in running cost
700+
Employees who kept working
2
Providers degraded at once
1
Endpoint in front of everything

The core idea: not every request deserves the most expensive model

The waste in most AI estates is that nobody sorts the work. Summarising a document, doing a first pass on a ticket, reformatting some text: all sent to the most expensive model available, because that is whatever the code was written against. The gateway's job is to have an opinion about that instead.

The unit that matters is the kind of work, not the application asking. Ten tools all doing summarisation are one kind of work, and they should all be priced and routed as one.

Request path
One endpoint, three decisions, then a tier
01
Any AI consumer calls one endpoint
Agents, internal tools and product features all speak to the gateway. None of them hold a provider key or name a model.
02
Policy classifies the work
What kind of work it is, how sensitive the data is, and whether each provider is healthy right now. Routed by task and by risk, never by habit.
03
The tier is selected
Frontier for high-stakes reasoning, small or self-hosted for volume, pinned self-hosted for regulated data.
04
Quality and cost are measured
Cost, speed, quality and how it fails, recorded for every kind of work against every model, so routing decisions rest on evidence.
Nothing that calls AI picks its own model. It describes the work, and one policy picks. That single inversion is what puts cost, data residency and failover in one place instead of in every codebase.
The tiers
What belongs where
Frontier tier
High-stakes reasoning and production agent decisions - the work where a subtle quality drop turns into a wrong action. Expensive per token, and worth it here.
Agent decisionsHigh-stakes reasoning
Volume tier
Classification, extraction, summarisation, first-pass analysis. Smaller or self-hosted open-weight models handle this at near-parity quality for a fraction of the cost.
ClassificationExtractionFirst-pass
Pinned self-hosted tier
Anything touching regulated data, pinned to self-hosted models entirely. This is what turned data-residency policy from aspiration into something the gateway enforces on every call.
Regulated dataResidency enforced
The tiering is the cost saving. Nothing else in the architecture matters as much as being honest about which work genuinely needs a frontier model.

Quality was measured, not assumed. Before any kind of work was moved to a cheaper model, we built a test set for it: twenty to fifty real examples with an agreed right answer. The cheaper model had to hold up against that set, and keep holding up, or the work went back.

The failover story

The outage day is the one that earned the gateway its budget.

Failover
Two providers degraded, nobody filed a ticket
State 01
Normal operation
Traffic split across frontier and volume tiers by policy. Self-hosted capacity runs warm, not idle.
State 02
Primary provider down
Automatic failover. Critical paths move to the secondary frontier provider; volume paths shift to self-hosted capacity.
State 03
Secondary degrades too
The self-hosted tier absorbs the remainder. Quality degrades gracefully instead of the service failing.
Outcome
Work continued
700+ employees worked through multi-hour outages at two providers. No tickets raised, because nothing they used stopped.
Resilience like this is not a retry in your code. It comes from nothing in the codebase naming a provider, routing that watches provider health, and enough of your own capacity to carry the critical paths when the market has a bad afternoon.

What it takes to build

Scope
Build these four things, and skip these two
Build
A thin routing layer in front of every AI consumer - one endpoint with policy attached
A test set for each kind of work, so 'good enough' is a defined, checkable bar before you route anything
Monitoring per route: cost, speed, quality and failure mode for each kind of work on each model
A kill switch per route, so a misbehaving model can be flipped back to a frontier provider in seconds
Do not
Turn it into a platform project - ours was a thin layer, not a programme
Self-host everything - self-host enough to cover critical paths and residency, no more
The test sets are the part teams skip, and then they wonder why quality got worse after a routing change. Build them before the routing, not after.
What the monitoring will tell you

Expect to find that a small minority of your work drives most of your bill. In our case roughly 20% of the kinds of work accounted for 80% of the spend, and it was visible within the first week. Until you can see cost broken down that way, every optimisation is guesswork.

The uncomfortable lessons

  1. The bill is a routing problem before it is a negotiation problem. We renegotiated contracts hard elsewhere (€600K a year off cloud security tooling at one point) - but AI cost fell mostly through architecture.
  2. "About as good" has to be measured. "Feels about the same" is not a migration strategy. Build the test set first, and keep running it.
  3. Self-hosted is a hedge, not a religion. We did not self-host everything. We self-hosted enough to make provider outages and data-residency rules someone else's emergency.
  4. Cost savings compound into agent fleets. The cheap tier is what made it affordable to run dozens of named agents continuously across more than forty connected systems.

Who should build one

If you are spending more than a few thousand a month on LLM APIs, or you have agents in production that a provider outage would blindside, a gateway pays for itself faster than almost anything else in the AI stack. It is also one of the highest-leverage things to get right before you scale agents - retrofitting routing into a live fleet is painful.

Do this next
A two-week path to your own gateway
1Pull last month's AI spend apart by kind of work rather than by application. The concentration you find is your business case.
2Pick your three highest-volume kinds of work and write a test set for each: twenty to fifty real examples with a right answer you could defend.
3Stand up one endpoint in front of one consumer, with two models behind it and the policy hard-coded. Prove the routing before you generalise it.
4Route one high-volume kind of work to a smaller or self-hosted model, and hold it against the test set for a week before extending.
5Write the residency rule into policy at the gateway, so regulated data cannot reach an external provider even by mistake.
6Add the kill switch and the per-route dashboard before you migrate anything that matters. Nobody trusts routing they cannot reverse.
Ask these
Questions for your next AI cost review
What did each kind of AI work cost us this month, and which three dominate the bill?
Which of our AI calls genuinely need the most expensive model, and who decided that, on what evidence?
If our primary provider went down for three hours tomorrow, what stops working, and who notices first?
Do any of our AI calls send regulated data to an external provider, and is that prevented or merely discouraged?
How would we detect a quality regression after a routing change - and would we detect it before a customer did?
How many places in our codebase hold a provider API key or a hard-coded model name?

We walk teams through this architecture regularly - including the guardrail and evaluation model around it. If it is relevant to you, an hour is usually enough to sketch your version of it.