When “Cheaper” LLM Hosting Gets Expensive

Author: Chitral Patil

“What does a token cost?”

Teams evaluating self-hosted large language models usually start with this simple question. It sounds like it should have a simple answer. In practice, it does not.

On the exact same GPU, running the exact same model, the real cost of a token can swing by as much as 36 times depending on one factor many estimates skip: how busy the hardware actually is.

That gap changes the build-versus-buy conversation. A self-hosted model can look inexpensive on paper while the underlying GPU is mostly idle and still billing by the hour. To make that cost visible, I built vllm-cost-meter, an open-source tool that attaches to a live vLLM server and reports the effective dollars-per-million-tokens for the traffic the server is actually handling.

vllm-cost-meter turns a live vLLM endpoint into a practical cost view, showing the effective cost per million tokens under the traffic the server is actually handling.

The input most calculators miss

Most LLM cost calculators start from a price per token, a hardware hourly rate, or a benchmark throughput number. Those inputs are useful, but they usually hide a major assumption: the GPU is fully utilized.

A GPU does not bill by useful work. It bills by time. If the hardware is busy, the fixed hourly cost spreads across many generated tokens. If the hardware is mostly idle, each token absorbs a larger share of that same fixed cost.

That means the effective cost of a token is not just a property of the model or the GPU. It is also a property of traffic.

What drives utilization is how many requests are in flight at once. That is driven by arrival rate and latency. A simple queueing relationship, Little’s Law, captures the intuition: the number of active requests is roughly the request arrival rate multiplied by how long each request takes. Low traffic means fewer requests in flight, a less utilized GPU, and a more expensive token. Higher sustained traffic fills the GPU and brings the effective cost down.

Same hardware. Same model. Different traffic. Different cost.

A quick analogy: rental car or rideshare?

Self-hosting an LLM resembles renting a car more than calling a rideshare.

For a one-off grocery run, a rideshare is usually cheaper because you only pay for the trip. For a two-week road trip, renting a car can be cheaper because the fixed daily cost gets spread across a lot of driving.

Self-hosted inference works the same way. You pay for the GPU whether it is busy or idle. Pay-per-token APIs behave more like the rideshare option: attractive for light or spiky usage, but potentially expensive at sustained high volume. Self-hosting can win when the hardware is kept busy, but the economics fall apart when the server handles only a trickle of requests.

The question is not simply whether self-hosting is cheaper. The question is whether your workload looks like a grocery run or a road trip.

What I measured

Rather than rely on idealized utilization assumptions, I ran controlled benchmark experiments across roughly one hundred runs. The experiments covered two GPU generations, multiple inference engines, several model families, and two numeric precisions. The goal was to measure the effective cost of output tokens at different traffic levels.

On identical H100 hardware:

• Effective cost ranged from $0.21 to $15.25 per million output tokens, a span of more than 70 times once different models and loads were included.
• At realistic low-to-moderate enterprise load, roughly 1 to 10 requests per second, underutilization created a 2.5 to 24 times cost penalty.
• Near idle, the penalty reached 36 times.

Effective cost per million output tokens versus offered load. As traffic rises, the fixed cost of the GPU spreads across more tokens and the per-token cost collapses. The expensive end of every curve is a near-idle, fully-priced GPU.

The curves all point to the same conclusion: the expensive end is not necessarily caused by a slow model or bad hardware. It is often caused by a fully priced GPU sitting mostly idle. I repeated the core sweep on A100 hardware as a cross-check, and the same load-driven effect appeared there as well, with a 7 to 11 times spread.

Why this matters for self-hosted inference

The practical rule is simple: if a cost estimate ignores utilization, it understates real cost by the inverse of utilization. A GPU running at 25 percent utilization makes the real cost per token roughly four times higher than a full-utilization estimate.

That error matters most for the workloads where self-hosting can look easiest to justify: low-traffic, internal, experimental, or bursty applications. Those workloads may not generate enough concurrent requests to keep the accelerator busy. A spreadsheet can make self-hosting look like an obvious win, while the live system tells a different story.

This is not an argument against self-hosting. It is an argument for measuring self-hosting against the workload you actually have.

What teams can do next

If you are evaluating self-hosted LLM inference, start here:

• Measure effective cost, not theoretical cost. Track dollars per million tokens using actual request traffic, not only benchmark throughput.
• Include utilization in build vs buy. A GPU hourly price is incomplete without the load profile that keeps the GPU busy.
• Test with realistic concurrency. Run experiments at the request rates and prompt shapes your application will actually see.
• Segment steady and bursty workloads. Put steady high-volume workloads on the shortlist for self-hosting. Treat spiky or low-volume workloads as candidates for pay-per-token APIs or shared infrastructure.
• Re-check the economics after launch. Traffic patterns change. A model that was cost-effective during a benchmark may be underutilized in production.

vllm-cost-meter makes this easier. Point it at a live vLLM endpoint, provide the hardware pricing assumptions, and measure the cost of tokens under real offered load. Under the hood, the tool formalizes the relationship between hardware, model, request shape, offered load, and latency target. As a user, the important part is simpler: it gives you a number tied to your server and your traffic.

The bottom line

The cost of self-hosting an LLM is not a single number you can look up. It depends on how hard you push the hardware.

Per-token API pricing makes cost visible because every request has a meter. Self-hosting can hide the meter inside an hourly infrastructure bill. If the GPU is busy, that can work in your favor. If it is idle, the true per-token cost can be many times higher than expected.

Before deciding that self-hosting is cheaper, measure it against real traffic. If the estimate still holds after utilization is included, the decision is much stronger. If it does not, it is better to know before the GPU bill arrives.

Chitral Patil is an Engineer II at GEICO, where he works on large language model inference and infrastructure. The work in this post is drawn from his paper Beyond Per-Token Pricing: A Concurrency-Aware Methodology for LLM Infrastructure Cost Estimation (arXiv:2606.11690) and the open-source cost meter vllm-cost-meter.