# Modalities

A modality is a kind of data: text, image, audio, document, video or vector. Every model reads text. Some also read images, audio or documents. The exchange can also buy images, speech, transcripts, video and embeddings.

## How it works

Most inputs use the chat endpoints: put a typed content part in a message. Image, speech, video and embedding jobs use their own endpoints.

| Modality         | You do                             | Endpoint                        | Page                                      |
| ---------------- | ---------------------------------- | ------------------------------- | ----------------------------------------- |
| Image in         | Send an image to a chat model      | `POST /v1/chat/completions`     | [Image input](https://inference.ai.exchange/docs/image-input.md)           |
| Audio in         | Send an audio clip to a chat model | `POST /v1/chat/completions`     | [Audio input](https://inference.ai.exchange/docs/audio-input.md)           |
| Document in      | Send a PDF to a chat model         | `POST /v1/chat/completions`     | [Document input](https://inference.ai.exchange/docs/document-input.md)     |
| Video in         | Send a video to a chat model       | `POST /v1/chat/completions`     | [Video input](#video-in)                  |
| Image out        | Buy an image, or get one in a chat | `POST /v1/images/generations`   | [Image generation](https://inference.ai.exchange/docs/image-generation.md) |
| Speech out       | Buy speech, or get it in a chat    | `POST /v1/audio/speech`         | [Speech](https://inference.ai.exchange/docs/speech.md)                     |
| Transcript out   | Buy a transcript of audio          | `POST /v1/audio/transcriptions` | [Speech-to-text](https://inference.ai.exchange/docs/speech-to-text.md)     |
| Video out        | Buy a generated video              | `POST /v1/videos`               | [Video generation](https://inference.ai.exchange/docs/video-generation.md) |
| Vectors out      | Buy embeddings for text            | `POST /v1/embeddings`           | [Embeddings](https://inference.ai.exchange/docs/embeddings.md)             |
| Reference inputs | Start a job from images or audio   | The job endpoints above         | [Reference inputs](https://inference.ai.exchange/docs/reference-inputs.md) |

An unchanged OpenAI client works for all of these. The Messages API takes images and documents, but no audio or video.

## Find a model that accepts a modality

A request with an attachment competes only among the offers that accept that kind of attachment, in the shape that you sent. To see what a model accepts, read `input_modalities` and `output_modalities` in `GET /v1/exchange/models` (no key needed), or open **Models → All models** in the console. An empty list means text only. Each provider sets its own limits, such as the largest image.

```json
{
  "model": "gpt-oss-120b",
  "input_modalities": [
    { "type": "text" },
    { "type": "image", "sources": ["url", "base64"], "max_bytes": 20000000 }
  ],
  "output_modalities": [{ "type": "text" }]
}
```

`GET /v1/exchange/models` gives one entry for each model and kind. `GET /v1/models` gives the cards that an SDK reads, with `kind`, `image_sizes` and `video_seconds`. Both accept these filters:

- `kind=image,speech,video`: one of `text`, `image`, `video`, `speech`, `transcription`, `embedding` or `decision`.
- `input=image,audio`: models that read every listed input kind.
- `output=audio`: models that write every listed output kind.
- Also `model`, `family`, `creator`, `feature`, `region`, `min_context`, `min_score`, `max_output_price`, `listed_within_days`, `contested`.

`/v1/exchange/models` refuses `provider`, `byok`, `cost_tier` and `max_input_price` with `400`. A bad value gets `400 invalid_model_filter`.

## Read a model card

`GET /v1/models` and `GET /v1/models/{id}` return one card for each model. It tells you what to send before you send it.

- `modality` is `text`, `image`, `speech`, `video`, `transcription`, `embedding` or `decision`.
- `kind` is the job kind. A chat model has none.
- `parameters` lists each control that at least one live offer supports. A name that is not a key is not supported, so omit it. Each key has a descriptor with a `type`:

| `type`    | Fields       | Example                                       |
| --------- | ------------ | --------------------------------------------- |
| `enum`    | `values`     | `size`: `["1024x1024", "1536x1024"]`          |
| `range`   | `min`, `max` | `seconds`: from 4 to 12, `output_compression` |
| `number`  | `min`, `max` | `creativity`: from 0 to 1                     |
| `integer` | none         | `seed`, `upscale_factor`                      |
| `boolean` | none         | `stream`, `turns`, `diarize`                  |
| `text`    | none         | `previous_job_id`                             |

- `pricing` lists the cheapest price of each billable: `billable`, `unit` (`mtok`, `image`, `character` or `second`), `usd` and, if the price depends on an option, the `variant`. A router model or a preset has no `pricing`.
- `image_sizes` and `video_seconds` list what image and video offers make. Image `resolution` and `aspect_ratio` are not in `parameters`. Read the tier and the shape from `image_sizes`.
- `liquid_supported_parameters` lists the feature names, such as `image.seed`.

```json
{
  "id": "acme-video-1",
  "modality": "video",
  "kind": "video_generation",
  "parameters": {
    "seconds": { "type": "range", "min": 1, "max": 12 },
    "resolution": { "type": "enum", "values": ["720p", "1080p"] },
    "seed": { "type": "integer" }
  },
  "pricing": [{ "billable": "output_video", "unit": "second", "usd": 0.05 }]
}
```

A control works only when an offer declares it. A request that sets a control with no offer for it gets `400 unsupported_parameter`, and `param` names the feature, such as `image.seed`. The exchange holds nothing and sends nothing.

## What you can send

| Kind     | Chat Completions part | Messages block | Accepted formats                                     |
| -------- | --------------------- | -------------- | ---------------------------------------------------- |
| Text     | `text`                | `text`         | Any text                                             |
| Image    | `image_url`           | `image`        | `image/png`, `image/jpeg`, `image/webp`, `image/gif` |
| Audio    | `input_audio`         | none           | `audio/wav`, `audio/flac`, `audio/mpeg`              |
| Document | `file`                | `document`     | `application/pdf`                                    |
| Video    | `video_url`           | none           | MP4, MOV and WebM, inline                            |

### URL or base64

| Kind     | `https://` URL | Base64 in a `data:` URL |
| -------- | -------------- | ----------------------- |
| Image    | Yes            | Yes                     |
| Audio    | No             | Yes                     |
| Document | No             | Yes                     |
| Video    | No             | Yes                     |

Only an image can be a remote `https://` URL. Send all other parts inline. The exchange refuses a remote document, audio or video part with `400 unsupported_content`.

The exchange does not fetch a URL, so it cannot measure a linked image. Send the bytes inline when you can: the cap is then exact, and more offers can serve the request.

**Size limit.** The body must fit the request limit, 20 MB by default. Base64 is one third larger than its bytes, so about 15 MiB of attachments fit. A larger body gets `413 request_too_large`.

## What an attachment costs

You pay for an attachment as input tokens. The price cap must cover it. The exchange measures each attachment from its header (PNG size, WAV sample rate, PDF page count) and decodes nothing. If it cannot read a header, it refuses the request with `400 unsupported_content`. It never prices a part at zero tokens.

Each offer converts a measured attachment to input tokens with its own table. If an offer states none, the exchange uses this one, which is set above the charge of each known upstream:

| Measured as             | Input tokens                    |
| ----------------------- | ------------------------------- |
| An image                | 85, plus 170 per 512-pixel tile |
| An image given as a URL | 1,200                           |
| Audio                   | 50 per second                   |
| Video                   | 300 per second                  |
| A PDF                   | 3,000 per page                  |

This table sets the cap and the ranking. The charge is the token count that the provider meters, so an attachment that the upstream read in fewer tokens costs less than the cap. See [Purchasing rules](https://inference.ai.exchange/docs/rules.md).

Image, speech and video jobs have a price per unit (one image, one character or one second of video), known before the job starts. For these jobs, the cap is exactly the charge.

## Video in

- Only a deployment that registered `video` input reads video. Otherwise a `video_url` part gets `400 unsupported_parameter` with `param` `input.video`.
- The exchange reads the length of an inline MP4, MOV or WebM file from its header and rounds it up to the second. The cap uses that length.
- A file with no total length in its header, such as a live stream or a fragmented MP4 with no `mehd` box, gets `400 unsupported_content`.

To make a video, see [Video generation](https://inference.ai.exchange/docs/video-generation.md).

## What the exchange does not do

The exchange does not convert between modalities. It does not transcribe audio, render a PDF as images, describe an image, or fetch, host or keep your files. A deployment that cannot read the kind that you sent cannot win.

## When no offer takes your request

The exchange refuses the request, and `eligibility.excluded` gives one reason for each offer.

- No offer for the model reads that kind of attachment: `400 unsupported_parameter`, with `param` naming the kind.
- Otherwise: `503 no_eligible_offer`.
- A part that no offer can serve is refused before the auction with `400 unsupported_content`. The error names the part, such as `messages[0].content[1]`. Examples: a remote document, audio or video; bad Base64; an unreadable header; an unknown part type.

The exchange never removes a part.

| Reason                | `detail`                                         | Meaning                                                |
| --------------------- | ------------------------------------------------ | ------------------------------------------------------ |
| `feature_unsupported` | `input.image`, `input.file`, …                   | The deployment does not read that kind at all.         |
| `media_unsupported`   | `image.max_bytes`, `image.formats: image/gif`, … | It reads that kind, but not in the shape of this part. |
| `job_kind_not_served` | none                                             | The deployment sells a different kind of job.          |

## Rules for image, speech and video jobs

These jobs have a price for each unit, so they follow different rules from chat jobs.

- **Router models.** A router model that ranks by price gets `400 unsupported_model`, because it ranks by token prices and text scores. A router model that ranks by Auto buys at the lowest estimated charge. A preset of another kind gets `400 unsupported_model`.
- **Token rules do not apply.** `exchange.limits.price.input_usd_per_mtok`, `exchange.limits.price.output_usd_per_mtok`, `exchange.limits.price.cache_read_usd_per_mtok`, `exchange.limits.price.cache_write_usd_per_mtok`, `cost_tier` other than `any`, and `exchange.limits.min_tps` get `400 unsupported_rule`, also from a preset. `cap_usd` and the other rules apply.
- **`exchange.limits.price.unit_usd`** sets the highest supplier price for one image, one character or one second of video. The exchange compares the charge of the job with this value times the count. An image offer priced in tokens is held to the charge of one image at its output token bound. A chat job gets `400 unsupported_rule`.