# How the exchange measures performance

The exchange measures the speed of each provider itself. It does not use the speed that a provider
claims. The measurements feed the predicted wait in [Price and wait](https://inference.ai.exchange/docs/price-and-wait.md) and the
speed ranking of `fastest` and [Auto Router](https://inference.ai.exchange/docs/auto.md).

## What the exchange measures

The exchange measures three numbers for each provider and each model.

| Measure           | What it tells you                                              | Jobs that count                                | Value of one job                                            |
| ----------------- | -------------------------------------------------------------- | ---------------------------------------------- | ----------------------------------------------------------- |
| **Base latency**  | The fixed time before the first token, in milliseconds.        | Jobs with fewer than 1,000 fresh input tokens. | Time from dispatch to the first token.                      |
| **Prefill speed** | How fast the provider reads a long prompt, in tokens a second. | Jobs with 16,000 or more fresh input tokens.   | Fresh input tokens ÷ time from dispatch to the first token. |
| **Output speed**  | How fast the provider writes the answer, in tokens a second.   | Jobs whose answer came as a stream.            | Output tokens ÷ time from the first token to the end.       |

**Fresh input tokens** are the input tokens that the provider had to process. They are the
uncached tokens plus the tokens written to the cache. Tokens read from the cache are not fresh.

A job's answer came as a stream when its output reached the router at four or more separate times,
each at least 10 ms after the one before. An answer that arrives in one piece has no output speed,
because the time from its first token to its end measures the network, not the model. On a receipt
from before the exchange counted these arrivals, the answer must take at least 250 ms instead.

Prefill speed counts all the wait before the first token. This includes the base latency, so the
number is a little lower than the true reading speed of the provider.

## Where the numbers come from

- The exchange takes the numbers from the receipts of settled jobs. Its own router records the
  times. A provider cannot change them.
- A time starts when the router sends the job to the provider. It ends when the first token
  arrives at the router. It includes the network and the queue of the provider.
- A job that failed and went to the next offer counts only for the offer that answered. The time of
  the failed attempt does not count.
- Probe jobs of the exchange count with real jobs. Synthetic test receipts do not count.

## How the numbers are combined

1. **Window.** The exchange uses the last seven complete UTC days. The newest day joins one hour
   after midnight, so that late receipts can arrive.
2. **Median.** The exchange takes the median of all jobs, not the average. One slow job does not
   move the number. Jobs are grouped in steps of 1%, so a value is correct to about 1%.
3. **Few jobs.** A provider with few jobs gets a value that is closer to the median of its model.
   - The weight of the provider's own value is `n ÷ (n + 10)`, where `n` is the number of jobs.
   - A provider with 10 jobs counts 50% of its own value. A provider with 90 jobs counts 90%.
4. **No jobs.** A provider with no jobs uses the median of its model. If the model has no data,
   the exchange uses the median of all models.
5. **Refresh.** The router reads the numbers once for each UTC day and keeps them in memory. If a
   read fails, the router keeps the last numbers.

## How the wait is predicted

For each offer, the exchange predicts the wait for your job:

```text
predicted wait = base latency
               + fresh input tokens ÷ prefill speed
               + expected output tokens ÷ output speed
```

- Fresh input tokens depend on your prompt size and on the cache. The exchange subtracts the tokens
  that it expects to read from the cache.
- The exchange estimates the output length of your job.
- If one of the three numbers is missing and the job needs it, the offer has no complete prediction.

## Where the measurements are used

| Use                                    | Measure that it reads                  | Rule                                                                                                         |
| -------------------------------------- | -------------------------------------- | ------------------------------------------------------------------------------------------------------------ |
| [Price and wait](https://inference.ai.exchange/docs/price-and-wait.md)  | Base latency, prefill and output speed | Adds the wait cost to the charge.                                                                            |
| `fastest` and [Auto Router](https://inference.ai.exchange/docs/auto.md) | Output speed                           | Ranks a provider by speed only after 5 jobs of its own. A value that came from other providers never counts. |

## Limits

- The numbers describe the last week. They do not show the load of a provider now.
- A new provider has no history. The exchange uses the model median until it has jobs.
- The exchange cannot tell network time from provider time. Both are in the numbers.
- Without a complete measurement, the exchange ranks on price alone for that auction.

## For providers

The `measured_p95_ttft_ms` field of your quote is not used in new decisions. To rank higher, answer
sooner and write faster. See [Provider quoting](https://inference.ai.exchange/docs/provider-quoting.md).

## Next steps

- [Price and wait](https://inference.ai.exchange/docs/price-and-wait.md)
- [Axis routing](https://inference.ai.exchange/docs/axis-routing.md)
- [Auto Router](https://inference.ai.exchange/docs/auto.md)