Don’t measure output. Measure useful output.
I want the industry to adopt the one benchmark that could disqualify our own architecture. It already exists. Reporting to it is optional.
Dr. Michael FörtschFounder and CEO, Q.ANT
We argue about accelerators in TOPS, in FLOPS, lately in tokens per second. Every one of those measures how fast a machine produces output. None of them measures whether the output was worth producing.
The metric already exists
MLPerf Inference fixes the model, the dataset and the quality target, then measures the AC power of the whole system at the wall: not the die, not a number from a datasheet. Completed work per second per watt, at a bar agreed before the comparison starts.
Two things follow
It works. Once the trade-off was visible, it narrowed. Vendors optimised for the thing they were being measured on. That is the whole argument for making the measurement standard rather than optional.
Almost nobody is compelled to report it. Procurement does not turn on it. So the industry quotes peak throughput and pays the electricity bill separately, as though the two were unrelated.
½of energy efficiency
Up to half the energy efficiency is spent on the last 0.9 points of accuracy, when systems move from a 99% to a 99.9% quality target.
Declaring an interest, and then the opposite of one
We sell photonic processors. Energy is our argument. But a benchmark with a hard accuracy bar is exactly the benchmark that puts an architecture like ours on trial. An efficiency advantage is cheap to claim and expensive to hold at a fixed quality target. I want that bar. Anyone who does not want it is telling you something.
And yes, “correct” is harder to define for a generative model than for a classifier. That difficulty is why we retreated to tokens per second. It is not a reason to stay there.
Which number is actually on the slide when your board approves the next tranche of compute?
