Workloads
A workload finds the cheapest model that does your task right. You give it the task and some cases with the answers that you expect. A run sends each case to up to four lanes, scores each answer field by field, and ranks the lanes by right answers and cost. A run never spends more than its limit. The default limit is $0.10.
An example
A team pulls four fields from each invoice email and needs 85% of the records right. It runs 120 cases on four lanes:
| Lane | Records right | $ / 1k right records |
|---|---|---|
liquid.auto |
93% | $0.37 |
| qwen3-32b | 88% | $0.24 |
deepseek-v4.1-flash on pera |
71% | $0.36 |
acme.cheap-json |
52% | $0.17 |
Auto gets the most records right, but qwen3-32b is also above the 85% floor and costs 35% less. The page names it as the cheapest lane at or above the floor. The numbers are an example.
When to use a workload
Use a workload for a task with one right answer, such as pulling fields from a document, labelling a ticket, or changing free text into a fixed JSON shape. Run it again when you change the prompt or a new model comes to the exchange.
Make your first workload
1. Write the task
- In the console, open Workloads and choose New workload.
- Give the workload a name.
- Write the system prompt and the user message. The user message must hold
{{ case.input }}. - Choose the model and the answer format.
- Choose Save as v1.
Or choose Save as workload in the playground. The last question becomes case 1.
2. Add cases
Choose New case, or Import a CSV or JSONL file. Start with 10 to 30 cases. More cases make each score range smaller. See Cases.
3. Run the lanes
A lane is one way to buy the answer: a provider of the model, another model, a preset or Auto. In the Runs tab, choose up to four lanes and the cases to send, then choose Run. See Lanes and takes.
4. Ship and watch
On the row of the lane that you want, choose Ship. Then choose Export as code and copy the code into your application. The Production tab shows its calls. See Watch the calls in production.
Read the result
A take is one answer of one lane to one case. A take is judged when its case has an expected answer and the exchange checked the answer against the format. Only judged takes count. A record is right when every field is right.
Lanes ranks the lanes of the newest run. The lane with the most right records is first. Among equal lanes, the lowest cost for 1,000 right records wins. A lane with no judged take goes last. Each lane shows:
- Records right, with a 95% confidence range.
- Fields right, and the share of answers that matched the schema.
- The cost of 1,000 takes and of 1,000 right records.
- The providers and models that served its takes.
The Empty answer row is the score of a model that answers nothing. A lane that does not beat it adds nothing. With a floor, a line above the lanes names the cheapest lane at or above it.
Other views: Fields (share right for each field and lane), History (the runs), and Wrong answers by field. In the run grid, select a square to see the answer, the expected answer and the receipt. The page shows an answer only for a judged take that is wrong.
Reference
The task and its versions
The task has a system prompt, a user message, an answer format, a model, the temperature, the output limit and the reasoning effort. If the task has no answer format and some cases have an expected answer, choose Draft from expected answers.
A save that changes the task makes a new version, such as Save as v2. A save that changes only the name does not.
Each version has a profile <workspace>.<workload>@v<n>. Each job of a run carries it as exchange.rank.workload_profile, and an offer can target it. See Purchasing rules.
These saves fail:
| Response | Cause |
|---|---|
409 slug_matches_key_profile |
A key of the workspace uses the workload name as the start of its default profile. |
400 invalid_request |
The profile starts with the slug of another workspace. |
Cases
Select a case to edit it. The expected answer has one field for each field of the answer format, and a blank field is null. A change to the cases saves the next case set and makes no version.
Import cases
- In the Cases column, choose Import.
- Choose a CSV or JSONL file of 25 MB or less.
- Choose the input column, and the expected answer column or None.
- For more than 200 rows, you can take a sample of 200 or 500 rows. The same file gives the same sample.
- Choose Keep to add to the cases, or Replace to remove them first. Then upload.
With an answer format, an expected answer must be JSON or a Python literal, such as {'name': None}. Otherwise it is the text of the cell. The page counts each row before the upload:
| Count | Meaning |
|---|---|
| No expected answer | A run cannot score the case. |
| All null | The expected answer holds only null. Finding nothing is right. |
| Left out: duplicate | The input is in the workload already, or earlier in the file. |
| Left out: breaks the format | The expected answer does not fit the answer format. |
| Left out: unreadable | The input is blank or over 32 KiB, or the answer cannot be read. |
The page uploads in batches, and each batch saves a case set. If the upload stops, the earlier batches stay in.
Lanes and takes
Add a provider of the model, or choose + Preset, model or Auto. Each take sends:
| Lane | The take sends |
|---|---|
| Provider | The model of the workload, and the provider in exchange.candidates.providers.allow. |
| Model | The name of the model as model. |
| Preset | The name of the preset as model. |
| Auto | The Auto name as model, with a price cap of $0.05 for each take. |
Auto refuses a request with no price bound, so the page adds the cap. See Auto.
A run uses the newest version and the newest case set, named in the Run task line. You also choose:
- The cases to send: 10, 30, 100, 300 or All, picked at random. When some cases have an expected answer, only those are sent.
- The takes of each case on each lane: 1, 3 or 5.
- The run limit. The default is $0.10.
Each take buys a new answer, because the response cache is off. Run is off when Setup has unsaved changes, the workload has no case, or a lane has no dry-run price yet.
Spend
A take starts only if the spend so far, the price caps of the takes in flight and its own price cap fit in the run limit. A take that does not fit waits and costs nothing. If you stop the run or reload the page, the waiting takes do not start.
How a take is scored
When a take ends, the page sends its job id and answer to the exchange. The exchange reads the receipt, scores the answer and keeps the result. It accepts only a job of your workspace that ran this version of this workload. If you close the page before it sends a take, that take is not scored. The line above the results counts these takes.
Scoring
The exchange compares each answer with the expected answer, field by field. A missing field is the same as null. Each field has a rule:
| Rule | A field is right when |
|---|---|
| Exact | It has the same value. This is the default. |
| Same text | It has the same text, in any case, with spaces and edge punctuation set aside. |
| Same set | It is a list with the same members, in any order. |
| Same set of texts | It is a list with the same members, in any order, compared as texts. |
Select An empty list and null are the same answer to count [] and null as equal. The Floor is the share of right records that a lane needs to be worth shipping. A change to the rules or the floor makes no version. To apply it to a run, score the run again.
Fix an expected answer
Choose Fix expected and correct the case. The fix saves the next case set. Choose Score on set to score the run again. A case whose input changed keeps its old score.
Ship a lane
Choose Ship on the row of a lane, or Ship v2 on this lane in the line that names the cheapest lane. Ship records the version and the lane, and shows them in the Ships column. It makes no version and changes no offer.
Watch the calls in production
The Production tab shows the calls that your code sends with a profile of the workload.
- Choose Export as code, which names the newest version, such as Export v2 as code.
- Copy the code in Python, TypeScript or curl, and send calls from your code. The exchange awards and charges each call as any other call.
- Read the result in the tab.
The code holds the prompts, the answer format and the settings of the newest version, a place for the input of each call, and exchange.rank.workload_profile. If you shipped a lane, it sends that lane as the model, with exchange.candidates.providers.allow or a price cap. When the shipped version is older, the tab says so.
The exchange compares each call that names a profile of the workload with the version, and marks the receipt. See Task match. Choose a window of 1, 7 or 30 days. The tab shows:
- Calls, Repeat the task (the share of marked calls that match the version), Schema pass and Spend.
- By key and Served by.
- Recent calls: the 20 newest, with version, mark, status, provider, charge and receipt.
The Production 7d column of the workload list shows the calls of the last 7 days.
What the exchange stores
A workload stores your prompt, the input and expected answer of each case, and the answer of each take. It is the only place where the exchange keeps request text. When you delete a workload, the exchange deletes all its versions, cases and runs. See Data handling.
Limits
50 workloads in a workspace. 5,000 cases in a workload. 32 KiB for one input. 25 MB for an import file. 4 lanes and 2,000 takes in a run. 8 takes run at one time.
Use a workload from the API
Send the profile of a version as exchange.rank.workload_profile on a request. See Workloads in the API. To rank offers by the chance of a good answer, see Fewest redos. The routes under /v1/workspaces/{slug}/workloads belong to the console and are not public API.