Cost per token is an outcome, not an input
Teams new to running models in production tend to ask the wrong first question: “what does inference cost per token?” The honest answer is that nobody sets that number directly. Cost per token is what falls out of a chain of underlying variables — how fast each GPU runs, how many requests it serves at once, how much of the hardware you actually keep busy, and how much you paid to rent or reserve it.
Get the chain right and the unit cost takes care of itself. Get it wrong and no amount of price negotiation saves you, because you are paying for hardware that sits idle or stalls under load. This piece walks the chain.
The three numbers that drive everything
Every serving system is a balancing act between latency, throughput, and utilization. They pull against each other, and the art is finding the operating point that fits your product.
Latency: time to first token
Time to first token (TTFT) is how long a user waits after sending a request before the first piece of the response appears. For interactive products — chat, copilots, agents — TTFT is the felt responsiveness of the whole experience. A model that is brilliant but slow to start feels broken.
TTFT is dominated by the prefill phase, where the model processes the prompt, and it is acutely sensitive to GPU memory bandwidth and to whether the request is competing for resources. On contended, multi-tenant hardware, TTFT becomes a moving target; on a dedicated inference endpoint, it becomes a specification you can hold.
Throughput: tokens per second
Tokens per second measures generation speed once decoding begins — both per-request (how fast one response streams) and aggregate (how much total work the cluster produces). Aggregate throughput is the lever that drives cost down: the more tokens a given GPU emits per second across all the requests it is handling, the cheaper each token becomes.
The tension is direct. Batching more requests together raises aggregate tokens per second and lowers cost per token, but it can raise latency for any individual request. Every serving configuration is a choice about where to sit on that curve.
Utilization: how much hardware you actually use
GPU utilization is the quiet killer of inference margins. A GPU you are paying for around the clock but keeping busy only part of the time is burning money the entire idle remainder. Real-world serving is bursty — traffic arrives in waves — so naive deployments leave accelerators starved between peaks.
This is where infrastructure stops being a commodity. Efficient batching, request routing, and the ability to pack a cluster’s work onto the GPUs you have are what turn raw silicon into low cost per token. Two teams can rent identical hardware and see cost-per-token figures that differ by a large multiple, purely because one runs its fleet hot and the other does not.
How the numbers combine into cost
The mental model is straightforward once the pieces are named:
- Your hardware bill is set by how many GPUs you hold and what you pay per GPU-hour (reserved capacity is cheaper per hour than on-demand; on-demand buys flexibility you pay a premium for).
- Your output is aggregate tokens per second, which is throughput per GPU times utilization across the fleet.
- Cost per token is the hardware bill divided by the tokens actually produced.
The implication is blunt: lowering cost per token is mostly an exercise in raising sustained tokens per second per dollar of committed hardware — which means raising utilization without blowing past your latency budget. Price shopping the hourly rate is the smallest lever in the equation.
Why infrastructure choices set the ceiling
You can tune a serving stack endlessly, but the infrastructure underneath sets the ceiling on what tuning can achieve.
- Tenancy. Shared fabric introduces variance you cannot engineer away. Single-tenant clusters give you a stable performance floor, which is what makes high, predictable utilization possible in the first place.
- Networking. For models sharded across GPUs, inter-GPU bandwidth and latency cap your achievable throughput. Low-latency fabrics are not a luxury here; they are the difference between a configuration that scales and one that stalls.
- Reliability. A failed accelerator that goes undetected does not just drop capacity — it drags utilization and latency across whatever was depending on it. Automated health checks and self-healing protect the economics, not just the uptime number.
Dedicated capacity versus a shared API
The cleanest expression of all this is the choice between a shared, per-call API and dedicated serving capacity. We break it down fully in dedicated inference vs API, but the economic logic is simple: a metered API is excellent at low or unpredictable volume, where you want someone else absorbing idle time. Past a certain steady throughput, you are paying a margin on every call that you could be capturing yourself by running dedicated capacity on a dedicated endpoint — with the added benefit that latency and data isolation stop being someone else’s variable.
The crossover point is a real, calculable number for your workload. Knowing where it sits is the difference between scaling on someone else’s margin and scaling on your own.
The takeaway
Inference economics is not a price list. It is a system in which latency, throughput, and utilization are set against each other on top of infrastructure that either raises or lowers the ceiling on all three. Evergrid’s single-tenant clusters exist to lift that ceiling — predictable performance, reserved capacity, and a cost profile you can forecast — so the unit economics of serving your model are something you engineer, not something you discover on the invoice.
