Modalities

A modality is a kind of data: text, image, audio, document, video or vector. Every model reads text. Some also read images, audio or documents. The exchange can also buy images, speech, transcripts, video and embeddings.

How it works

Most inputs use the chat endpoints: put a typed content part in a message. Image, speech, video and embedding jobs use their own endpoints.

An unchanged OpenAI client works for all of these. The Messages API takes images and documents, but no audio or video.

Find a model that accepts a modality

A request with an attachment competes only among the offers that accept that kind of attachment, in the shape that you sent. To see what a model accepts, read input_modalities and output_modalities in GET /v1/exchange/models (no key needed), or open Models → All models in the console. An empty list means text only. Each provider sets its own limits, such as the largest image.

{
  "model": "gpt-oss-120b",
  "input_modalities": [
    { "type": "text" },
    { "type": "image", "sources": ["url", "base64"], "max_bytes": 20000000 }
  ],
  "output_modalities": [{ "type": "text" }]
}

GET /v1/exchange/models gives one entry for each model and kind. GET /v1/models gives the cards that an SDK reads, with kind, image_sizes and video_seconds. Both accept these filters:

• kind=image,speech,video: one of text, image, video, speech, transcription, embedding or decision.
• input=image,audio: models that read every listed input kind.
• output=audio: models that write every listed output kind.
• Also model, family, creator, feature, region, min_context, min_score, max_output_price, listed_within_days, contested.

/v1/exchange/models refuses provider, byok, cost_tier and max_input_price with 400. A bad value gets 400 invalid_model_filter.

Read a model card

GET /v1/models and GET /v1/models/{id} return one card for each model. It tells you what to send before you send it.

• modality is text, image, speech, video, transcription, embedding or decision.
• kind is the job kind. A chat model has none.
• parameters lists each control that at least one live offer supports. A name that is not a key is not supported, so omit it. Each key has a descriptor with a type:

• pricing lists the cheapest price of each billable: billable, unit (mtok, image, character or second), usd and, if the price depends on an option, the variant. A router model or a preset has no pricing.
• image_sizes and video_seconds list what image and video offers make. Image resolution and aspect_ratio are not in parameters. Read the tier and the shape from image_sizes.
• liquid_supported_parameters lists the feature names, such as image.seed.

{
  "id": "acme-video-1",
  "modality": "video",
  "kind": "video_generation",
  "parameters": {
    "seconds": { "type": "range", "min": 1, "max": 12 },
    "resolution": { "type": "enum", "values": ["720p", "1080p"] },
    "seed": { "type": "integer" }
  },
  "pricing": [{ "billable": "output_video", "unit": "second", "usd": 0.05 }]
}

A control works only when an offer declares it. A request that sets a control with no offer for it gets 400 unsupported_parameter, and param names the feature, such as image.seed. The exchange holds nothing and sends nothing.

What you can send

URL or base64

Only an image can be a remote https:// URL. Send all other parts inline. The exchange refuses a remote document, audio or video part with 400 unsupported_content.

The exchange does not fetch a URL, so it cannot measure a linked image. Send the bytes inline when you can: the cap is then exact, and more offers can serve the request.

Size limit. The body must fit the request limit, 20 MB by default. Base64 is one third larger than its bytes, so about 15 MiB of attachments fit. A larger body gets 413 request_too_large.

What an attachment costs

You pay for an attachment as input tokens. The price cap must cover it. The exchange measures each attachment from its header (PNG size, WAV sample rate, PDF page count) and decodes nothing. If it cannot read a header, it refuses the request with 400 unsupported_content. It never prices a part at zero tokens.

Each offer converts a measured attachment to input tokens with its own table. If an offer states none, the exchange uses this one, which is set above the charge of each known upstream:

This table sets the cap and the ranking. The charge is the token count that the provider meters, so an attachment that the upstream read in fewer tokens costs less than the cap. See Purchasing rules.

Image, speech and video jobs have a price per unit (one image, one character or one second of video), known before the job starts. For these jobs, the cap is exactly the charge.

Video in

• Only a deployment that registered video input reads video. Otherwise a video_url part gets 400 unsupported_parameter with param input.video.
• The exchange reads the length of an inline MP4, MOV or WebM file from its header and rounds it up to the second. The cap uses that length.
• A file with no total length in its header, such as a live stream or a fragmented MP4 with no mehd box, gets 400 unsupported_content.

To make a video, see Video generation.

What the exchange does not do

The exchange does not convert between modalities. It does not transcribe audio, render a PDF as images, describe an image, or fetch, host or keep your files. A deployment that cannot read the kind that you sent cannot win.

When no offer takes your request

The exchange refuses the request, and eligibility.excluded gives one reason for each offer.

• No offer for the model reads that kind of attachment: 400 unsupported_parameter, with param naming the kind.
• Otherwise: 503 no_eligible_offer.
• A part that no offer can serve is refused before the auction with 400 unsupported_content. The error names the part, such as messages[0].content[1]. Examples: a remote document, audio or video; bad Base64; an unreadable header; an unknown part type.

The exchange never removes a part.

Rules for image, speech and video jobs

These jobs have a price for each unit, so they follow different rules from chat jobs.

• Router models. A router model that ranks by price gets 400 unsupported_model, because it ranks by token prices and text scores. A router model that ranks by Auto buys at the lowest estimated charge. A preset of another kind gets 400 unsupported_model.
• Token rules do not apply. exchange.limits.price.input_usd_per_mtok, exchange.limits.price.output_usd_per_mtok, exchange.limits.price.cache_read_usd_per_mtok, exchange.limits.price.cache_write_usd_per_mtok, cost_tier other than any, and exchange.limits.min_tps get 400 unsupported_rule, also from a preset. cap_usd and the other rules apply.
• exchange.limits.price.unit_usd sets the highest supplier price for one image, one character or one second of video. The exchange compares the charge of the job with this value times the count. An image offer priced in tokens is held to the charge of one image at its output token bound. A chat job gets 400 unsupported_rule.