The question is not whether a hosted API token is more expensive than a GPU hour. It is whether your traffic keeps that GPU busy. Self-hosting an open model with vLLM beats a hosted API only past a throughput break-even, and the variable that moves the break-even is utilization, not the per-token price you see on a pricing page. Most teams compare list prices, pick the cheaper unit, and miss that a rented H100 costs the same whether it serves 5 requests per second or sits idle overnight.
What actually drives the cost
A hosted API has zero fixed cost and a fixed marginal cost: you pay per token, the provider absorbs idle capacity, and your bill scales linearly from request one. Self-hosting inverts that. You pay for the GPU by the hour regardless of load, so your real cost per token is the hourly rate divided by tokens actually generated in that hour. The lever that fills the denominator is continuous batching, vLLM's core mechanism. Instead of waiting for a batch to assemble, it admits new sequences into the running batch at each decode step and evicts finished ones, so the GPU keeps generating while requests arrive and complete at different times. Paged attention keeps the KV cache from fragmenting, which is what lets you pack more concurrent sequences onto one card.
This is why the same hardware can swing an order of magnitude in cost per token. At 90% utilization an H100 is cheap per token. At 10% it is far more expensive than any hosted API. You are not buying tokens, you are buying GPU-hours and converting them to tokens at an efficiency you control.
The break-even math
Treat both options as cost per million output tokens and find where they cross. Indicative figures, round numbers you should replace with your own quotes:
- Rented H100, roughly 2 to 3 USD per hour on a committed plan. Serving an 8B-class model with vLLM at healthy batch sizes, sustained throughput sits in the low thousands of output tokens per second.
- At 2,000 tokens/sec, one card produces about 7.2M tokens per hour, so 2.50 USD/hour is roughly 0.35 USD per million output tokens, fully utilized.
- A comparable hosted small model runs on the order of 0.50 to 1.00 USD per million output tokens with no idle risk and no ops.
Fully utilized, self-hosting wins by 2 to 3x. But that 0.35 figure assumes the card never idles. Effective cost is list cost divided by utilization: at 30% utilization the same card costs about 1.15 USD per million, and the hosted API is now cheaper. The crossover lands near 30 to 50% sustained utilization for an 8B model on one GPU. Below it, hosted wins. The practical threshold: you need enough steady traffic to keep at least one card busy through the trough of your daily curve, not just the peak.
When not to self-host
The break-even is necessary, not sufficient. Skip self-hosting when traffic is spiky or low-volume, because a card sized for peak idles through the average and your real utilization collapses. Skip it when you need frontier reasoning quality that open weights do not yet match, when latency on cold or bursty traffic matters more than unit cost, or when you have no one to own GPU drivers, model upgrades, autoscaling, and the 3am page. The hosted premium is buying capacity elasticity and an on-call team. Below a few hundred million tokens a month, that premium is almost always worth paying.
A hybrid routing pattern
You rarely have to choose globally. Route per request. Run a vLLM fleet sized to your traffic floor, the volume you sustain through the trough, and burst the overflow to a hosted API. A thin router in front decides per call:
- Default to the self-hosted open model for the high-volume, well-scoped work: classification, extraction, summarization, RAG answer generation.
- Spill to the hosted API when the local queue depth or time-to-first-token crosses a threshold, so a traffic spike never tail-latencies your users while you wait on autoscale.
- Route to a frontier hosted model for the small slice of requests that genuinely need it, flagged by task type or a difficulty signal, not by default.
This pins your base load to the cheap, high-utilization path and pays the hosted premium only on the burst and the hard cases. The router needs a uniform request schema across backends, a real queue-depth or TTFT signal from vLLM rather than a fixed concurrency cap, and per-route logging so you can watch where tokens and money actually go.
Why it matters
In our engagements the open-model fleet earns its keep only when a workload is steady and high-volume enough to keep the GPUs busy, and the hybrid split keeps the hosted bill confined to overflow and genuinely hard requests. Decide on measured utilization, not list price. Instrument tokens per second per card, sustained utilization, and per-route cost before you commit capital to GPUs, then size the fleet to the floor and let the API absorb the rest.
