How the exchange measures performance

The exchange measures the speed of each provider itself. It does not use the speed that a provider claims. The measurements feed the predicted wait in Price and wait and the speed ranking of fastest and Auto Router.

What the exchange measures

The exchange measures three numbers for each provider and each model.

Measure	What it tells you	Jobs that count	Value of one job
Base latency	The fixed time before the first token, in milliseconds.	Jobs with fewer than 1,000 fresh input tokens.	Time from dispatch to the first token.
Prefill speed	How fast the provider reads a long prompt, in tokens a second.	Jobs with 16,000 or more fresh input tokens.	Fresh input tokens ÷ time from dispatch to the first token.
Output speed	How fast the provider writes the answer, in tokens a second.	Jobs whose answer came as a stream.	Output tokens ÷ time from the first token to the end.

Fresh input tokens are the input tokens that the provider had to process. They are the uncached tokens plus the tokens written to the cache. Tokens read from the cache are not fresh.

A job's answer came as a stream when its output reached the router at four or more separate times, each at least 10 ms after the one before. An answer that arrives in one piece has no output speed, because the time from its first token to its end measures the network, not the model. On a receipt from before the exchange counted these arrivals, the answer must take at least 250 ms instead.

Prefill speed counts all the wait before the first token. This includes the base latency, so the number is a little lower than the true reading speed of the provider.

Where the numbers come from

The exchange takes the numbers from the receipts of settled jobs. Its own router records the times. A provider cannot change them.
A time starts when the router sends the job to the provider. It ends when the first token arrives at the router. It includes the network and the queue of the provider.
A job that failed and went to the next offer counts only for the offer that answered. The time of the failed attempt does not count.
Probe jobs of the exchange count with real jobs. Synthetic test receipts do not count.

How the numbers are combined

Window. The exchange uses the last seven complete UTC days. The newest day joins one hour after midnight, so that late receipts can arrive.
Median. The exchange takes the median of all jobs, not the average. One slow job does not move the number. Jobs are grouped in steps of 1%, so a value is correct to about 1%.
Few jobs. A provider with few jobs gets a value that is closer to the median of its model.
- The weight of the provider's own value is n ÷ (n + 10), where n is the number of jobs.
- A provider with 10 jobs counts 50% of its own value. A provider with 90 jobs counts 90%.
No jobs. A provider with no jobs uses the median of its model. If the model has no data, the exchange uses the median of all models.
Refresh. The router reads the numbers once for each UTC day and keeps them in memory. If a read fails, the router keeps the last numbers.

How the wait is predicted

For each offer, the exchange predicts the wait for your job:

predicted wait = base latency
               + fresh input tokens ÷ prefill speed
               + expected output tokens ÷ output speed

Fresh input tokens depend on your prompt size and on the cache. The exchange subtracts the tokens that it expects to read from the cache.
The exchange estimates the output length of your job.
If one of the three numbers is missing and the job needs it, the offer has no complete prediction.

Where the measurements are used

Use	Measure that it reads	Rule
Price and wait	Base latency, prefill and output speed	Adds the wait cost to the charge.
`fastest` and Auto Router	Output speed	Ranks a provider by speed only after 5 jobs of its own. A value that came from other providers never counts.

Limits

The numbers describe the last week. They do not show the load of a provider now.
A new provider has no history. The exchange uses the model median until it has jobs.
The exchange cannot tell network time from provider time. Both are in the numbers.
Without a complete measurement, the exchange ranks on price alone for that auction.

For providers

The measured_p95_ttft_ms field of your quote is not used in new decisions. To rank higher, answer sooner and write faster. See Provider quoting.