# Workloads

A workload finds the cheapest model that does your task right. You give it the task and some cases with the answers that you expect. A run sends each case to up to four lanes, scores each answer field by field, and ranks the lanes by right answers and cost. A run never spends more than its limit. The default limit is $0.10.

## An example

A team pulls four fields from each invoice email and needs 85% of the records right. It runs 120 cases on four lanes:

| Lane                          | Records right | $ / 1k right records |
| ----------------------------- | ------------- | -------------------- |
| `liquid.auto`                 | 93%           | $0.37                |
| qwen3-32b                     | 88%           | $0.24                |
| deepseek-v4.1-flash on `pera` | 71%           | $0.36                |
| `acme.cheap-json`             | 52%           | $0.17                |

Auto gets the most records right, but qwen3-32b is also above the 85% floor and costs 35% less. The page names it as the cheapest lane at or above the floor. The numbers are an example.

## When to use a workload

Use a workload for a task with one right answer, such as pulling fields from a document, labelling a ticket, or changing free text into a fixed JSON shape. Run it again when you change the prompt or a new model comes to the exchange.

## Make your first workload

### 1. Write the task

1. In the console, open **Workloads** and choose **New workload**.
2. Give the workload a name.
3. Write the system prompt and the user message. The user message must hold `{{ case.input }}`.
4. Choose the model and the [answer format](https://inference.ai.exchange/docs/structured-output.md).
5. Choose **Save as v1**.

Or choose **Save as workload** in the playground. The last question becomes case 1.

### 2. Add cases

Choose **New case**, or **Import** a CSV or JSONL file. Start with 10 to 30 cases. More cases make each score range smaller. See [Cases](#cases).

### 3. Run the lanes

A lane is one way to buy the answer: a provider of the model, another model, a preset or Auto. In the **Runs** tab, choose up to four lanes and the cases to send, then choose **Run**. See [Lanes and takes](#lanes-and-takes).

### 4. Ship and watch

On the row of the lane that you want, choose **Ship**. Then choose **Export as code** and copy the code into your application. The **Production** tab shows its calls. See [Watch the calls in production](#watch-the-calls-in-production).

## Read the result

A take is one answer of one lane to one case. A take is **judged** when its case has an expected answer and the exchange checked the answer against the format. Only judged takes count. A record is **right** when every field is right.

**Lanes** ranks the lanes of the newest run. The lane with the most right records is first. Among equal lanes, the lowest cost for 1,000 right records wins. A lane with no judged take goes last. Each lane shows:

- **Records right**, with a 95% confidence range.
- **Fields right**, and the share of answers that matched the schema.
- The cost of 1,000 takes and of 1,000 right records.
- The providers and models that served its takes.

The **Empty answer** row is the score of a model that answers nothing. A lane that does not beat it adds nothing. With a [floor](#scoring), a line above the lanes names the cheapest lane at or above it.

Other views: **Fields** (share right for each field and lane), **History** (the runs), and **Wrong answers by field**. In the run grid, select a square to see the answer, the expected answer and the receipt. The page shows an answer only for a judged take that is wrong.

## Reference

### The task and its versions

The task has a system prompt, a user message, an answer format, a model, the temperature, the output limit and the reasoning effort. If the task has no answer format and some cases have an expected answer, choose **Draft from expected answers**.

A save that changes the task makes a new version, such as **Save as v2**. A save that changes only the name does not.

Each version has a profile `<workspace>.<workload>@v<n>`. Each job of a run carries it as `exchange.rank.workload_profile`, and an offer can target it. See [Purchasing rules](https://inference.ai.exchange/docs/rules.md).

These saves fail:

| Response                       | Cause                                                                              |
| ------------------------------ | ---------------------------------------------------------------------------------- |
| `409 slug_matches_key_profile` | A key of the workspace uses the workload name as the start of its default profile. |
| `400 invalid_request`          | The profile starts with the slug of another workspace.                             |

### Cases

Select a case to edit it. The expected answer has one field for each field of the answer format, and a blank field is `null`. A change to the cases saves the next **case set** and makes no version.

### Import cases

1. In the Cases column, choose **Import**.
2. Choose a CSV or JSONL file of 25 MB or less.
3. Choose the input column, and the expected answer column or **None**.
4. For more than 200 rows, you can take a sample of 200 or 500 rows. The same file gives the same sample.
5. Choose **Keep** to add to the cases, or **Replace** to remove them first. Then upload.

With an answer format, an expected answer must be JSON or a Python literal, such as `{'name': None}`. Otherwise it is the text of the cell. The page counts each row before the upload:

| Count                       | Meaning                                                          |
| --------------------------- | ---------------------------------------------------------------- |
| No expected answer          | A run cannot score the case.                                     |
| All null                    | The expected answer holds only `null`. Finding nothing is right. |
| Left out: duplicate         | The input is in the workload already, or earlier in the file.    |
| Left out: breaks the format | The expected answer does not fit the answer format.              |
| Left out: unreadable        | The input is blank or over 32 KiB, or the answer cannot be read. |

The page uploads in batches, and each batch saves a case set. If the upload stops, the earlier batches stay in.

### Lanes and takes

Add a provider of the model, or choose **+ Preset, model or Auto**. Each take sends:

| Lane     | The take sends                                                                        |
| -------- | ------------------------------------------------------------------------------------- |
| Provider | The model of the workload, and the provider in `exchange.candidates.providers.allow`. |
| Model    | The name of the model as `model`.                                                     |
| Preset   | The name of the preset as `model`.                                                    |
| Auto     | The Auto name as `model`, with a price cap of $0.05 for each take.                    |

Auto refuses a request with no price bound, so the page adds the cap. See [Auto](https://inference.ai.exchange/docs/auto.md).

A run uses the newest version and the newest case set, named in the **Run task** line. You also choose:

- The cases to send: 10, 30, 100, 300 or **All**, picked at random. When some cases have an expected answer, only those are sent.
- The takes of each case on each lane: 1, 3 or 5.
- The run limit. The default is $0.10.

Each take buys a new answer, because the response cache is off. **Run** is off when **Setup** has unsaved changes, the workload has no case, or a lane has no dry-run price yet.

### Spend

A take starts only if the spend so far, the price caps of the takes in flight and its own price cap fit in the run limit. A take that does not fit waits and costs nothing. If you stop the run or reload the page, the waiting takes do not start.

### How a take is scored

When a take ends, the page sends its job id and answer to the exchange. The exchange reads the receipt, scores the answer and keeps the result. It accepts only a job of your workspace that ran this version of this workload. If you close the page before it sends a take, that take is not scored. The line above the results counts these takes.

### Scoring

The exchange compares each answer with the expected answer, field by field. A missing field is the same as `null`. Each field has a rule:

| Rule              | A field is right when                                                          |
| ----------------- | ------------------------------------------------------------------------------ |
| Exact             | It has the same value. This is the default.                                    |
| Same text         | It has the same text, in any case, with spaces and edge punctuation set aside. |
| Same set          | It is a list with the same members, in any order.                              |
| Same set of texts | It is a list with the same members, in any order, compared as texts.           |

Select **An empty list and null are the same answer** to count `[]` and `null` as equal. The **Floor** is the share of right records that a lane needs to be worth shipping. A change to the rules or the floor makes no version. To apply it to a run, score the run again.

### Fix an expected answer

Choose **Fix expected** and correct the case. The fix saves the next case set. Choose **Score on set** to score the run again. A case whose input changed keeps its old score.

### Ship a lane

Choose **Ship** on the row of a lane, or **Ship v2 on this lane** in the line that names the cheapest lane. Ship records the version and the lane, and shows them in the **Ships** column. It makes no version and changes no offer.

### Watch the calls in production

The **Production** tab shows the calls that your code sends with a profile of the workload.

1. Choose **Export as code**, which names the newest version, such as **Export v2 as code**.
2. Copy the code in Python, TypeScript or curl, and send calls from your code. The exchange awards and charges each call as any other call.
3. Read the result in the tab.

The code holds the prompts, the answer format and the settings of the newest version, a place for the input of each call, and `exchange.rank.workload_profile`. If you shipped a lane, it sends that lane as the `model`, with `exchange.candidates.providers.allow` or a price cap. When the shipped version is older, the tab says so.

The exchange compares each call that names a profile of the workload with the version, and marks the receipt. See [Task match](https://inference.ai.exchange/docs/workloads-api.md#task-match). Choose a window of 1, 7 or 30 days. The tab shows:

- **Calls**, **Repeat the task** (the share of marked calls that match the version), **Schema pass** and **Spend**.
- **By key** and **Served by**.
- **Recent calls**: the 20 newest, with version, mark, status, provider, charge and receipt.

The **Production 7d** column of the workload list shows the calls of the last 7 days.

### What the exchange stores

A workload stores your prompt, the input and expected answer of each case, and the answer of each take. It is the only place where the exchange keeps request text. When you delete a workload, the exchange deletes all its versions, cases and runs. See [Data handling](https://inference.ai.exchange/docs/data-handling.md).

### Limits

50 workloads in a workspace. 5,000 cases in a workload. 32 KiB for one input. 25 MB for an import file. 4 lanes and 2,000 takes in a run. 8 takes run at one time.

### Use a workload from the API

Send the profile of a version as `exchange.rank.workload_profile` on a request. See [Workloads in the API](https://inference.ai.exchange/docs/workloads-api.md). To rank offers by the chance of a good answer, see [Fewest redos](https://inference.ai.exchange/docs/rules.md#fewest-redos). The routes under `/v1/workspaces/{slug}/workloads` belong to the console and are not public API.