The Tokenomics of AI: Why “Cloud vs. Self-Hosted” Is the Wrong Question

Most organisations are asking whether to self-host their AI or rent it from the cloud. That framing is already costing them money. The real discipline (call it tokenomics) is matching each task to the cheapest infrastructure that still clears its bar for quality, latency, and data sovereignty. Here's how to think about it in 2026.

There's a seductive story going around boardrooms: open-weight models have caught up, GPUs are cheaper, so surely the smart move is to pull AI in-house and stop paying the cloud tax.

Half of that story is true. The open-weight models (Llama, Qwen, Mistral, DeepSeek) have closed most of the quality gap, landing within a few percentage points of frontier closed models on the majority of practical benchmarks. Inference costs have fallen 40–60% since 2024. The tooling (vLLM, TGI, Ollama, llama.cpp) is genuinely production-grade.

The other half is where organisations lose money. Because "self-host to save money" is, for most workloads, simply false. The teams that learn this after signing the GPU lease learn it expensively.

Let me lay out how I actually think about this.

The break-even math is brutal, and most teams are on the wrong side of it

Self-hosting only beats a cloud API on cost when two things are true at once: you're comparing against an expensive frontier model, and you're running at high, predictable volume.

The thresholds are sobering. Against premium frontier APIs, self-hosting tends to break even somewhere around 5–10 million tokens a month, a bar a serious production system can clear. But against the cheap end of the cloud market, specifically the budget and mini-tier models now priced at cents per million tokens, the break-even can sit north of 100 million tokens a day, a volume most organisations will never touch. At that point the GPU would need to run at near-total utilisation around the clock just to match the API bill.

And the GPU rental is the floor, not the ceiling. The costs that sink TCO projections are the ones that don't appear on the pricing page:

• Engineering time. A production self-hosted stack needs real DevOps/MLOps attention: deploying, tuning batch sizes and KV cache, patching CUDA, responding to incidents. Budget 10–20 hours a month at minimum; at enterprise scale, a dedicated team of 4–6 engineers running $700K–$1.5M a year.
• The model treadmill. New model versions land every couple of months. Each one is an evaluation-and-migration project you now own.
• The physical tail. Power and cooling (a high-end GPU under sustained load adds 25–40% on top just for cooling), networking egress, storage for weights and logs, and N+1 redundancy so a single dead GPU doesn't take your service down.

The uncomfortable summary: outside of specific conditions, managed cloud APIs are cheaper than self-hosting for the large majority of enterprise use cases once you count the full stack. Anyone spinning up an H100 cluster and calling it "cost optimisation" should show their working.

A worked example: the same workload, before and after

Abstractions are easy to nod along to and hard to act on, so here's a concrete one.

Picture an internal assistant that answers questions over your own content, the kind of thing most organisations are now standing up. Say it handles 100,000 queries a month. Each query pulls in a chunk of context and returns a short answer, roughly 8,000 input tokens and 500 output tokens apiece.

The old setup: one flagship model for everything. It's the natural first instinct: pick the best model on the market and point everything at it. Simple to build, and every answer is top-quality. At frontier pricing (~$5 per million input tokens, ~$25 per million output), the bill lands around $5,250 a month (about $63,000 a year).

Here's the thing nobody checks: the overwhelming majority of those 100,000 queries are simple. "What's our policy on X?" "Summarise this document." A budget model answers those indistinguishably from a flagship one. You're paying Michelin prices to boil an egg — 90,000 times a month.

The new setup: route each task to the cheapest model that clears its bar. Same queries, same quality target, one change: a routing layer that reads the request and sends it to the right tier.

• 70% are simple → a budget model at ~$0.15/$0.60 per million tokens. Cost: about $105.
• 20% are moderate → a mid-tier model at ~$3/$15. Cost: about $630.
• 10% are genuinely hard → the frontier model, where it earns its price. Cost: about $525.

New total: ~$1,260 a month (about $15,120 a year). Same output quality, a 76% smaller bill, from a routing layer, not a rebuild.

The second decision: stop paying for the same context twice. Look at what's actually in those 8,000 input tokens. Roughly half of every request is identical every time: the system prompt, the instructions, the tool definitions, the standing context the assistant always needs. You're sending it, and being billed for it, on all 100,000 queries, even though the model has already processed that exact block thousands of times.

Prompt caching fixes exactly this. The provider stores the processed state of a stable prefix and bills repeat hits at a 90% discount (0.1× the normal input rate on current flagship models). You pay a small one-time premium to "warm" the cache, then near-nothing every time that prefix recurs. At this volume (~3,300 queries a day keeps the cache continuously warm) the stable half of every prompt effectively drops off the bill.

Fold that in and the number moves again: ~$882 a month (about $10,600 a year). That's another 30% off the routed figure, and ~83% below where we started, with, to be clear, zero change to a single answer the user sees.

That's the whole play. No migration, no data centre: two decisions (route by difficulty, cache the repeats) that together take a $63,000 bill to about $10,600. And notice the discipline underneath both: you only ever spend where the task actually demands it.

One honest caveat before anyone over-extrapolates the arrow: caching only ever discounts input tokens: output is never cached. Any savings estimate that applies the discount to your whole bill is wrong. It's also exact-prefix matching, so a single moving part in your stable block (a timestamp, a session ID, a stray newline) silently drops your hit rate to zero. The savings are real, but they're an engineering discipline, not a switch.

The rest of the toolkit: every lever, and when it actually pays

Routing and caching are the two biggest levers, which is why they carry the example. A mature setup stacks several more on top. Here's the fuller toolkit, with the mechanic and the catch on each, because every one of these has a failure mode that turns the saving into a loss.

Batch processing: 50% off, if you can wait. Both major providers run an asynchronous "batch" tier at half the standard rate for work that doesn't need an answer this second: overnight document processing, bulk classification, embedding generation, scheduled report runs. The catch is latency: results return within a window (often up to 24 hours), so it's useless for anything user-facing. The elegant part is that it stacks with caching: a batch call against a cached prefix can land near 95% off the list input rate (half off, then 90% off the cached portion). If a workload is both repetitive and non-urgent and you're not batching it, you're leaving money on the table.

Context compression: the biggest lever in agentic workloads. The per-token price isn't what makes agents expensive; the accumulated context is. Every tool call in an agent loop re-sends the entire conversation history plus retrieved files, so by mid-session you're paying frontier rates on 80,000+ tokens the model has already seen. Compressing that context before it hits the model (summarising old turns, trimming retrieved chunks to what's relevant, dropping stale tool output) routinely cuts input tokens 40–70% with no quality loss, and on agent-heavy stacks it dwarfs everything else. The catch: compression takes judgement (or a cheap model to do the summarising), and over-aggressive trimming quietly degrades answers.

Semantic caching: the one that touches output too. Prompt caching only discounts the input side of an exact prefix match. Semantic caching lives in your application layer and does something different: it stores full responses and serves them when a new query is semantically similar to one already answered, bypassing the model entirely, saving both input and output. For workloads with heavy question overlap (support, internal FAQs, the same research questions asked a hundred ways) hit rates can be high and each hit is nearly free. The catch is correctness: "similar" isn't "identical," so you need a conservatively tuned threshold and a way to bound the error rate, or you'll serve a subtly wrong answer with complete confidence.

Output discipline: the side of the bill nobody caches. Output tokens cost 4–10× more than input, and no caching scheme ever discounts them. That makes generation length the most under-managed cost in most stacks. Capping max_tokens, prompting for terse answers, and using structured output instead of prose cut output spend meaningfully on high-volume tasks: a classifier returning one label shouldn't be free-writing a paragraph. The catch is trivial but real: don't starve the tasks that genuinely need long answers.

Fine-tuning and distillation: move volume down a tier for good. If a specific high-volume task keeps hitting a frontier model, a smaller open model fine-tuned or distilled on that narrow task can often match the big model's quality on that task alone, and the whole slice then moves to a cheap tier permanently. This is how you shrink the expensive 10% from the example over time. The catch: upfront cost and ownership. You now maintain a model artefact, its evaluation, and its retraining as the task drifts.

Quantization: only relevant once you self-host. If a workload does cross into self-hosted territory, running the model at reduced precision (4-bit or 8-bit) cuts its memory footprint 2–4× for typically 1–3% quality loss, often letting one cheaper GPU serve a model that would otherwise need two. In 2026 a well-quantized 32B model handles what needed a 70B a couple of years ago. The catch: the quality hit is task-dependent, so measure it on your workload, not a leaderboard.

Provider arbitrage: the same model, a different price. Open-weight models are served by multiple hosted-inference providers, and the same weights vary meaningfully in price and speed between them, purely on infrastructure efficiency. A model-agnostic architecture (which you want anyway) lets you route to the cheapest capable endpoint and switch when the market moves, which it does every quarter. The catch is operational surface: more providers, more keys, more monitoring.

The thread through all of it: none is a blanket win. Each pays off under specific conditions and quietly costs you under others. Which is why the closing instruction matters more than any single technique.

So when should you self-host? When the reason isn't cost.

Notice that everything so far (routing, caching, and the whole toolkit above) never once reached for a self-hosted GPU. That's deliberate. At these volumes, the budget cloud tier is already so close to the floor that self-hosting the high-volume slice wouldn't beat it on cost. So when does self-hosting actually enter the picture? When the reason has nothing to do with the per-token price:

• Data sovereignty. This is the big one. If a workload touches data you cannot (legally or strategically) send to a third party, the API price is irrelevant. Regulated industries (health, finance, legal) often have no other compliant path. And any organisation whose core asset is proprietary data has a strategic version of the same concern.
• Predictable, high-volume, latency-sensitive workloads. Steady baseline traffic where you control the hardware and shave off the network round-trip.
• Proprietary fine-tunes. If you've fine-tuned an open model on your own data and it beats the general-purpose APIs for your use case, self-hosting is the only way to run it.

Notice that none of these is "it's cheaper." They're about control, compliance, and specialisation.

The actual answer is a portfolio, not a decision

The framing that saves money isn't "cloud or self-hosted." It's a deliberate hybrid, with each request routed to the cheapest backend that clears its bar. Organisations that get this right report 30–70% savings against a naïve all-cloud stack, not by picking a side, but by refusing to.

The routing logic is simple to state:

1. Commodity, high-volume, non-sensitive tasks (classification, extraction, routine drafting) go to cheap models, self-hosted if the volume justifies it, or a budget cloud tier if not.
2. Frontier-intelligence tasks (the hard 5% that genuinely need the best reasoning) go to a top-tier cloud model. You don't build a data centre for the 5%.
3. Anything touching your crown-jewel data goes to a private endpoint or a self-hosted model, regardless of what the cost spreadsheet says.

Three things make this practical in 2026 and didn't a couple of years ago:

1. A managed middle ground exists. Bedrock, Vertex, Azure OpenAI and the like serve frontier models from inside your cloud boundary (private endpoints, VNet integration, enterprise SLAs) for roughly a 15–40% markup over raw API pricing. For most regulated enterprises, that's the right starting point, not full self-hosting.
2. Zero-data-retention API tiers are now standard at the major frontier labs, with contractual guarantees that inputs aren't retained or trained on. For many organisations that closes the privacy gap without a single GPU. The real question becomes: do you trust a contractual guarantee, or does your threat model demand physical control? That's a risk decision, not a technology one.
3. Model-agnostic architecture is cheap insurance. Build on an abstraction layer (LiteLLM, Portkey, or similar) and standard protocols like MCP so you can swap any model (cloud or local) without rewriting the system around it. The single most expensive mistake is hard-wiring your stack to one provider and losing the ability to route.

The one instruction I’d give any leadership team

Instrument first. Decide second.

Almost every organisation I see arguing about self-hosting is arguing from estimated token volumes, not measured ones. Start on cloud APIs (where you pay nothing for idle hardware and get frontier access on day one) and measure your actual usage by task type. The moment a specific, predictable, high-volume workload crosses its break-even line, move that one workload to self-hosted infrastructure. Not the whole estate. That workload.

Tokenomics isn't a philosophy about where AI "should" live. It's an operating discipline: know your volumes, know your data-sensitivity map, and route every token to the cheapest place that can be trusted to handle it. The organisations that internalise this now (that treat inference like a portfolio to be optimised rather than a side to be chosen) will hold a compounding advantage in both cost and control for years.

The rest will keep paying flagship prices for commodity tasks, or running half-idle GPU clusters they mistook for savings. The gap between those two groups is going to get very wide, very fast.

This piece is published by the SystemBender engineering team. It grew out of a conversation about local model hosting that started, of all places, on a tennis court.