Nvidia believes AI infrastructure should be judged on cost per token: a metric where the new Blackwell GPUs reportedly score exceptionally well.
Nvidia criticized in a blog post that companies are using the wrong metric to evaluate AI infrastructure. Instead of looking at the cost per GPU hour or FLOPS per dollar, the real benchmark should be cost per token. This is the total cost to produce one million AI tokens.
According to Nvidia, data centers have since evolved into “AI factories” that no longer just provide computing power, but primarily “produce intelligence in the form of tokens.” Therefore, Nvidia argues it is irrelevant to judge infrastructure solely on pure performance. It is more accurate to look at the economic efficiency of AI workloads.
Tokens are important there. These are small pieces of data that AI models work with to generate text, code, or other output. You can see them as a kind of unit for AI thinking and as an economic metric for AI. The faster and cheaper tokens are processed, the more AI yields.
Internal kitchen
The company reinforces this claim with a comparison between its own Hopper and Blackwell generation GPUs. However, users still have to wait for the latter due to supply chain issues. Per GPU, Blackwell costs about twice as much as Hopper. If you look at FLOPS per dollar, Blackwell is twice as efficient. But the token output tells a different story: Blackwell delivers 65 times more tokens per second per GPU, 50 times more tokens per megawatt, and the cost per million tokens is 35 times lower ($0.12 compared to $4.20).
According to Nvidia, token costs must therefore go down to maximize revenue. This can be achieved by increasing the delivered token output. But how is that token output determined? Nvidia describes it as an ‘inference iceberg.’ The price to use a GPU for an hour is the visible part above water. What lies beneath, from software to network architecture, determines how many tokens a system delivers per second, and thus what each token would ultimately cost.
Source: Nvidia
Token costs as the new standard
If you compare AI infrastructure based on computing costs or FLOPS per dollar, Nvidia argues you get an incomplete picture of inference costs. The focus on cost per token is logical, as it allows Nvidia to demonstrate how efficient the new GPUs are and that they can enable low-cost AI architectures.
In effect, Nvidia is changing the narrative. Where hardware used to be judged based on an objective performance metric (FLOPS) and the power consumption linked to it, Nvidia now wants to convince organizations to look at tokens per watt.
There is some merit to that, but as Nvidia itself says, the rest of the hardware and software stack is also relevant to the number of processed tokens, not just the GPU itself. The fact that Nvidia provides an entire software ecosystem and offers its GPUs in total solutions with its own CPUs (Vera Rubin and DPUs) fits perfectly into that story.
Tokens per watt are thus not an objective metric for comparing the pure performance of a GPU, but they can offer insight into the efficiency of a total AI system. It is therefore not surprising that Nvidia, as a provider of a fully integrated stack, prefers tokens as the unit of measurement.