Prompt caching

Prompt caching makes a repeated prompt start cheaper. A multi-turn chat or an agent loop gets the saving with no setup.

Many prompts start the same way: a system prompt, tool definitions or a long document. A provider can store that start, called the prefix. A later request with the same prefix reads it from the cache, and the provider charges a fraction of the input rate for those tokens. The exchange makes a cache plan for each job and sends it to the winning provider.

To store a whole answer at the exchange and skip the provider, see Response caching.

What a cache costs

• A read costs less than input, typically 0.1 to 0.5 times the input rate.
• A write stores a prefix. A five-minute write typically costs 1 to 1.25 times the input rate. A one-hour write costs about 2 times. Many providers write for free.

The first request that sends a prefix pays for a write. Each later request that reuses the prefix pays for a read. The lifetime of a write is its time-to-live (TTL). After the TTL ends, the next request writes again.

Example: input at USD 3.00 for each million tokens, a read at 0.1 times, a five-minute write at 1.25 times, and a 10,000-token prefix. The rates are an illustration.

The write pays back at the first read. If no later request reads the prefix, the write is a loss, so auto does not cache a one-off request.

You pay the rates of the winning offer. The order book shows the cache-read and cache-write rates of each offer. A provider that quotes no cache-read rate gets no cache saving in the ranking.

Check what a job saved

Each response has an exchange block. Its charge object has three cache fields:

• total_usd: what you paid.
• cache_usd: the part of the charge, fee included, for cache reads and writes.
• uncached_usd: what you would have paid with no cache, at the same rates.

cache_usd and uncached_usd are present only when some input used a cache, and no money moves on them. A job that hit its price cap has no cache_usd. If uncached_usd is smaller than total_usd, a write was never read.

The price cap includes cache writes up to the last breakpoint. It does not include an assumed saving, and a write cannot charge more than the cap.

The usage block reports cached input separately from fresh input:

On Chat Completions both are inside prompt_tokens_details, and prompt_tokens counts them. On Messages, input_tokens excludes them: add the three for the total input.

The receipt of the job shows the cache tokens and the charge. If an offer quotes no write rate, the exchange bills written tokens as uncached input.

If the second request shows no read

Check these causes:

• The prefix changed before the breakpoint, even by one character.
• The TTL ended between the two requests.
• The prompt is shorter than the minimum of the upstream. See What each upstream caches.
• Another provider won the second request. See Which provider serves the next turn.

Choose how to cache

Set exchange.cache.mode in the request body to choose who places the breakpoints. A breakpoint is a point in the prompt. The provider caches everything before it.

Let the exchange cache (auto)

If the request declares tools or continues a conversation, the exchange places breakpoints after the tool definitions, after the system prompt, after the previous request, and after the last message. The next request reads the last one.

It places breakpoints only if the estimated prompt is 512 tokens or more, the smallest minimum of the Anthropic models. It does not cache a one-off request. Its breakpoints always use the five-minute TTL. Use your own markers to ask for one hour.

Place your own breakpoints (auto or buyer)

Add cache_control to the last block of the prefix to cache. Under auto, a request with any marker keeps your markers and gets none from the exchange. Under buyer, the exchange never places its own.

Any ttl other than "1h" is five minutes. Use the short write for requests within five minutes of each other, such as one agent loop. Use the long write when requests are further apart. A long write pays back only if the prefix is read after the short TTL would have ended.

Messages API (Anthropic format). Mark a block of system, a tool, or a block inside a turn:

import os
from anthropic import Anthropic

client = Anthropic(
    base_url="https://router.inference.ai.exchange",
    api_key=os.environ["LIQUID_API_KEY"],
)

response = client.messages.create(
    model="liquid.auto",
    max_tokens=1024,
    system=[
        {"type": "text", "text": "You are terse."},
        {
            "type": "text",
            "text": long_reference_document,
            "cache_control": {"type": "ephemeral", "ttl": "1h"},  # long write
        },
    ],
    messages=[{"role": "user", "content": "What changed in section 4?"}],
)
print(response.usage)

Chat Completions. Mark a content part or a tool. A marker needs the array form of content:

response = client.chat.completions.create(
    model="liquid.auto",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": long_reference_document,
                    "cache_control": {"type": "ephemeral"},  # short write
                }
            ],
        },
        {"role": "user", "content": "What changed in section 4?"},
    ],
)

Rules for markers:

• A request can have four breakpoints at most. The exchange drops the shallowest extra ones.

• Anthropic refuses a one-hour breakpoint after a five-minute one, so each breakpoint before a one-hour breakpoint also becomes one hour.

• A marker on a tool puts the breakpoint after the last tool.

• A top-level cache_control object puts one breakpoint after the last message, in both APIs. A value that is not an object places nothing:

  response = client.chat.completions.create(
      model="liquid.auto",
      messages=messages,
      extra_body={"cache_control": {"type": "ephemeral"}},
  )
  

Switch off the cache (off)

off drops your own markers and the cache key. It cannot stop an upstream that caches long prompts itself.

response = client.chat.completions.create(
    model="liquid.auto",
    messages=[{"role": "user", "content": "hello"}],
    extra_body={"exchange": {"cache": {"mode": "off"}}},
)

Keep a conversation on one cache

Name the conversation

Send session_id in the request body, or the x-session-id header, with the same value on each turn. The body field wins. The value can have 256 characters at most.

The exchange picks the cache key in this order: session_id, prompt_cache_key, then a digest of the first block of the prompt. Some providers, such as OpenAI, route by this key. The exchange salts the key for each buyer, so it never joins the requests of two buyers. With exchange.cache.mode: "off", the exchange ignores session_id.

response = client.chat.completions.create(
    model="liquid.auto",
    messages=messages,
    extra_body={"session_id": "support-chat-4821"},
)

Which provider serves the next turn

A cache lives at one provider. The exchange remembers which provider served the last turn of a session, for as long as the longest cache that the turn wrote. Each new turn renews the record.

For that provider, the ranking estimate assumes a cache read for part of the prompt, but only if the provider publishes a cache-read rate.

• The assumed share is the measured hit rate of the deployment over 30 days of settled jobs, or one half with too little history.
• To set the share yourself, use exchange.cache.expected_hit_rate in Purchasing rules.

When the cache saves anything, the exchange also ranks that provider 5% below its own estimate. So the provider that served the last turn wins unless another offer is cheaper by more than the cache saving plus that margin. The price cap does not include the assumed saving. Under Auto, a session can also stay on the previous deployment. See Auto Router.

A new conversation can also find a warm cache. The exchange remembers, for each buyer, which provider served each prefix, for the same time as a session. This works without a session_id, and the exchange can lose a record early.

Keep a session on one deployment

Set exchange.session.sticky: true in the exchange block or in a preset. The deployment that served the previous turn then ranks first while it is eligible: it passes your rules and its price cap fits the cap of the call. Otherwise the exchange ranks as usual. A fallback to the next offer happens only before any output.

response = client.chat.completions.create(
    model="liquid.auto",
    messages=messages,
    extra_body={"session_id": "support-chat-4821", "exchange": {"session": {"sticky": True}}},
)

exchange.session.sticky is not the Auto session pin of Auto versions 5 and 6. Set exchange.session.pin: "off" to turn the pin off, or exchange.session.margin to change the score points that a rival must add. See Session pin.

Privacy between buyers

Only you can read your cached prompts. The exchange puts a code that is unique to your account before your system prompt and messages in each cacheable prompt, so a shared upstream account never shares a prefix between buyers. One exception: some upstreams cache the tool definitions before the system prompt, so buyers can share the cache for an identical list of tools.

Reference

The meters

The exchange meters the input of a job in four parts and the output as a fifth. Each part has its own rate on the offer.

The public offers in Market data carry the rates in the cache_read, cache_write_short and cache_write_long lines of pricing. A line is absent when the offer quotes no such price.

What each upstream caches

The exchange sends the same cache plan to each provider, in the form that its upstream reads. The upstream decides what it caches and what it charges. The multipliers below are list prices of the upstream, and an offer can quote other rates. A prompt shorter than the minimum of the upstream is never cached and is billed as uncached input.

Anthropic

• You choose where the cache starts, with cache_control markers or exchange breakpoints.
• A five-minute write costs 1.25 times the input rate, a one-hour write 2 times, and a read 0.1 times (less on some newer models).
• The minimum cacheable prompt depends on the model, from 512 to 4,096 tokens. A prompt below it is processed without a cache and with no error, also with a marker. See the Anthropic documentation for the minimum of each model.
• The cache checks at most 20 blocks back from each breakpoint. In a long turn, place a new breakpoint.
• A change to the tool definitions invalidates the tools, the system prompt and the messages. A change to the system prompt invalidates the system prompt and the messages.

OpenAI

• OpenAI caches a long prefix by itself. On GPT-5.6 and later, the minimum is 1,024 visible input tokens.
• On GPT-5.6 and later, a write costs 1.25 times the input rate and a read 0.1 times. Earlier models have no write charge.
• The exchange bills written tokens at the write rate of the offer only when the request has a cache breakpoint. Otherwise it bills them as uncached input.
• The exchange sends prompt_cache_key, and OpenAI routes requests with the same key to the same cache. Keep the start of the prompt the same and put changing text at the end.

Other upstreams

Groq, Grok, Moonshot AI, DeepSeek, Z.AI, Google Gemini and others cache a repeated prefix by themselves. The exchange sends the same body as for OpenAI, and the upstream can ignore the key.

• A read costs less than input. A write is usually free, and the exchange then bills written tokens as uncached input.
• DeepSeek reports cached input as prompt_cache_hit_tokens. The exchange bills these tokens at the cache-read rate of the offer.
• Alibaba caches only at cache_control markers, with a five-minute TTL, and charges for the write.
• The exchange does not create explicit Gemini caches. Put stable text first.

These rules are a guide. The usage block shows what the provider reported, and the receipt shows what you paid.

Next steps

• Response caching: store a whole answer and get it back at no charge.
• Purchasing rules: the exchange.cache.mode setting and exchange.session.sticky.