The AI industry has converged on a deceptively simple metric: cost per token. It’s easy to understand, easy to compare, and easy to market. Every new system promises to drive it lower. Charts show steady declines, sometimes dramatic ones, reinforcing the impression that AI inference is rapidly becoming cheaper and more efficient.
But simplicity, in this case, is misleading.
A token is not a fundamental unit of cost in isolation. It is the visible output of a deeply complex system that spans model architecture, hardware design, system scaling, memory behavior, power consumption, and operational efficiency. Reducing that complexity to a single number creates a dangerous illusion: improvements in cost per token necessarily reflect improvements in the underlying system.
They often do not.
To understand what is really happening, we need to step back and look at the full system—specifically, the total cost of ownership (TCO) of an AI inference deployment.
From benchmark numbers to real systems
Most comparisons in the industry start from benchmark results. Inference benchmarks such as MLPerf provide a useful baseline because they fix key variables—model, latency constraints, and workload characteristics—allowing different systems to be evaluated under the same conditions.
Take a large-scale model such as Llama 3.1 405B. On a modern GPU system like Nvidia’s GB200 NVL72, MLPerf reports an aggregate throughput that translates to roughly 138 tokens per second per accelerator. An alternative inference-focused architecture might deliver a lower figure—say, 111 tokens per second per accelerator.
At first glance, the conclusion seems obvious: the GPU is faster.
But this is precisely where the problem begins. That number describes the performance of a single accelerator under specific benchmark conditions. It says very little about how the system behaves when deployed at scale.
And in real-world data centers, scale is everything.
The illusion of linear scaling
In theory, performance should scale linearly with the number of accelerators. Double the hardware, double the throughput. In practice, this never happens. Communication overhead, synchronization, memory contention, and architectural inefficiencies all conspire to reduce effective performance as systems grow.
This effect is captured by what is often called scaling efficiency. It’s one of the most important and most overlooked parameters in AI infrastructure.
A system that achieves 97% scaling efficiency will behave differently from one that achieves 85%, even if their per-chip performance appears comparable. Over dozens or hundreds of accelerators, that difference compounds rapidly.
This is where inference-specific architectures begin to separate themselves.
Unlike training, inference does not require backpropagation. The execution flow is more predictable, the data movement patterns are more structured, and the opportunity for optimization is significantly greater. Architectures that are purpose-built for inference can exploit this determinism to sustain high utilization across large systems.
One architecture is a case in point. By moving away from the traditional GPU execution model and adopting a deeply pipelined, dataflow-oriented design, it minimizes the coordination overhead that typically erodes scaling efficiency. The result is not just higher peak utilization, but more important, consistently high utilization at scale.
When the system flips the narrative
Once performance is evaluated at the level that actually matters—servers, racks, and data centers—the comparison often changes.
Throughput per server depends not only on per-accelerator performance, but also on how many accelerators are packed into a system and how efficiently they work together. Throughput per rack adds another layer, incorporating system density and infrastructure constraints. When power is introduced into the equation, the relevant metric becomes throughput per kilowatt.
It is at this level that architectural differences become impossible to ignore.
GPU-based systems are optimized for flexibility. They can handle a wide range of workloads, but that generality introduces inefficiencies when running highly structured inference tasks. Data must move between memory hierarchies, threads must be synchronized, and execution units often sit idle waiting for dependencies to resolve.
The architecture mentioned above takes a different approach. By eliminating the traditional memory hierarchy bottlenecks and replacing them with a large, flat register file combined with a dataflow execution model, it effectively removes the “memory wall” that limits sustained performance in GPU systems. Data is kept close to compute, and execution proceeds in a continuous pipeline rather than in discrete, synchronized steps.
The consequence is subtle but powerful: even if peak per-chip performance appears lower, the effective throughput at the system level can be significantly higher. More importantly, that performance is achieved with far greater energy efficiency.
Power: The constraint that doesn’t go away
Energy consumption is not just a cost factor; it’s the constraint that ultimately defines the scalability of AI infrastructure.
Electricity prices, power usage effectiveness (PUE), and utilization rates are not theoretical constructs. They are operational realities that directly impact the economics of every deployment. A system that consumes less energy per token has an intrinsic advantage that compounds over time.
This is where inference-native architectures again demonstrate their value.
Because the architecture’s design minimizes unnecessary data movement and maximizes pipeline utilization, it delivers more tokens per unit of energy. The metric that matters is not peak FLOPS, but tokens per kilowatt—and on that axis, architectural efficiency becomes the dominant factor.
In large-scale deployments, this translates directly into lower operating costs and improved total cost of ownership.
The hidden influence of workload assumptions
Benchmarking does not eliminate bias—it simply moves it.
Parameters such as context length, output token size, and concurrency have a profound impact on system behavior. A model running at 128K context imposes different demands than one operating at 8K. Latency, memory pressure, and throughput all shift accordingly.
Architectures that rely on heavy memory movement are particularly sensitive to these changes. As context length grows, the cost of moving data becomes increasingly dominant.
By contrast, architectures that localize data and streamline execution are more resilient to these shifts. This is another area where the architecture’s register-centric, dataflow design provides an advantage: it reduces dependence on external memory bandwidth and maintains more consistent performance across varying workloads.
From metrics to economics
When performance, power, and infrastructure are combined, the discussion moves from engineering to economics.
Total cost of ownership captures the full picture: capital expenditure, operating costs, energy consumption, and system utilization over time. It reflects not just how fast a system can run, but how efficiently it can deliver value in a real deployment.
This is where many cost-per-token claims fall apart.
A lower cost per token can be achieved in multiple ways—by improving efficiency, by adjusting assumptions, or by accepting lower margins. Without a system-level view, it’s impossible to distinguish between these scenarios.
What matters is not the headline number, but the underlying drivers.
The risk of optimizing the wrong thing
The industry’s focus on cost per token has created a subtle distortion. Instead of optimizing systems, we risk optimizing metrics. This is not unique to AI. Every technology cycle has its preferred metrics, and every metric can be gamed if taken out of context.
A truly efficient system is one that aligns performance, energy consumption, and scalability. It delivers consistent throughput, minimizes waste, and operates effectively under real-world constraints. This is precisely the direction that inference-specific architectures are taking.
The aforementioned architectural approach illustrates this shift. Rather than attempting to adapt a general-purpose architecture to an increasingly specialized workload, it starts from the workload itself and builds upward. The result is a system that is not only efficient in theory, but also in practice—at scale, under load, and within the constraints of real data centers.
Toward a more honest conversation
None of this diminishes the achievements of GPU-based systems. They have been instrumental in the rise of modern AI and remain incredibly powerful platforms. But the workloads are changing. Large language model inference is not the same as training, and it’s not the same as graphics. As the industry shifts toward deployment at scale, the limitations of general-purpose architectures become more apparent.
At the same time, new architectures, as described above, are emerging that are designed specifically for these workloads. They may not always win on peak performance metrics, but they are optimized for the realities of inference: predictable execution, high utilization, and energy efficiency.
If we want to compare these systems fairly, we need to move beyond simplified metrics and toward system-level evaluation.
The bottom line
Cost per token is not wrong—but it is incomplete.
The real question is not how cheaply a token can be produced in isolation, but how efficiently a system can deliver tokens over time, at scale, within the constraints of power, infrastructure, and workload demands.
When viewed through that lens, the path forward becomes clearer.
The next generation of AI infrastructure will not be defined by the highest peak performance or the most aggressive benchmark result. It will be defined by architectures that align performance with efficiency, and efficiency with economics.
And in that context, the industry may find that the most important innovation is not faster hardware—but better architecture.
Lauro Rizzatti is a business development executive at VSORA, a pioneering technology company offering silicon semiconductor solutions that redefine performance. He is a noted chip design verification consultant and industry expert on hardware emulation.
Related Content