# Prompt caching

Prompt caching makes a repeated prompt start cheaper. A multi-turn chat or an agent loop gets the saving with no setup.

Many prompts start the same way: a system prompt, tool definitions or a long document. A provider can store that start, called the prefix. A later request with the same prefix reads it from the cache, and the provider charges a fraction of the input rate for those tokens. The exchange makes a cache plan for each job and sends it to the winning provider.

To store a whole answer at the exchange and skip the provider, see [Response caching](https://inference.ai.exchange/docs/response-caching.md).

## What a cache costs

- A **read** costs less than input, typically 0.1 to 0.5 times the input rate.
- A **write** stores a prefix. A five-minute write typically costs 1 to 1.25 times the input rate. A one-hour write costs about 2 times. Many providers write for free.

The first request that sends a prefix pays for a write. Each later request that reuses the prefix pays for a read. The lifetime of a write is its time-to-live (TTL). After the TTL ends, the next request writes again.

Example: input at USD 3.00 for each million tokens, a read at 0.1 times, a five-minute write at 1.25 times, and a 10,000-token prefix. The rates are an illustration.

| Request         | Without a cache | With a cache |
| --------------- | --------------- | ------------ |
| 1 (write)       | USD 0.0300      | USD 0.0375   |
| 2 (read)        | USD 0.0300      | USD 0.0030   |
| 3 (read)        | USD 0.0300      | USD 0.0030   |
| Total for three | USD 0.0900      | USD 0.0435   |

The write pays back at the first read. If no later request reads the prefix, the write is a loss, so `auto` does not cache a one-off request.

You pay the rates of the winning offer. The order book shows the cache-read and cache-write rates of each offer. A provider that quotes no cache-read rate gets no cache saving in the ranking.

## Check what a job saved

Each response has an `exchange` block. Its `charge` object has three cache fields:

- `total_usd`: what you paid.
- `cache_usd`: the part of the charge, fee included, for cache reads and writes.
- `uncached_usd`: what you would have paid with no cache, at the same rates.

`cache_usd` and `uncached_usd` are present only when some input used a cache, and no money moves on them. A job that hit its price cap has no `cache_usd`. If `uncached_usd` is smaller than `total_usd`, a write was never read.

The price cap includes cache writes up to the last breakpoint. It does not include an assumed saving, and a write cannot charge more than the cap.

The `usage` block reports cached input separately from fresh input:

| API              | Cache read                | Cache write                   |
| ---------------- | ------------------------- | ----------------------------- |
| Chat Completions | `cached_tokens`           | `cache_write_tokens`          |
| Messages         | `cache_read_input_tokens` | `cache_creation_input_tokens` |

On Chat Completions both are inside `prompt_tokens_details`, and `prompt_tokens` counts them. On Messages, `input_tokens` excludes them: add the three for the total input.

The receipt of the job shows the cache tokens and the charge. If an offer quotes no write rate, the exchange bills written tokens as uncached input.

### If the second request shows no read

Check these causes:

- The prefix changed before the breakpoint, even by one character.
- The TTL ended between the two requests.
- The prompt is shorter than the minimum of the upstream. See [What each upstream caches](#what-each-upstream-caches).
- Another provider won the second request. See [Which provider serves the next turn](#which-provider-serves-the-next-turn).

## Choose how to cache

Set `exchange.cache.mode` in the request body to choose who places the breakpoints. A breakpoint is a point in the prompt. The provider caches everything before it.

| Value            | Who places the breakpoints                                        |
| ---------------- | ----------------------------------------------------------------- |
| `auto` (default) | Your `cache_control` markers, or the exchange when you sent none. |
| `buyer`          | Only your own `cache_control` markers.                            |
| `off`            | No one. The exchange sends no breakpoints and no cache key.       |

### Let the exchange cache (`auto`)

If the request declares tools or continues a conversation, the exchange places breakpoints after the tool definitions, after the system prompt, after the previous request, and after the last message. The next request reads the last one.

It places breakpoints only if the estimated prompt is 512 tokens or more, the smallest minimum of the Anthropic models. It does not cache a one-off request. Its breakpoints always use the five-minute TTL. Use your own markers to ask for one hour.

### Place your own breakpoints (`auto` or `buyer`)

Add `cache_control` to the last block of the prefix to cache. Under `auto`, a request with any marker keeps your markers and gets none from the exchange. Under `buyer`, the exchange never places its own.

| Marker                               | TTL       | Meter               |
| ------------------------------------ | --------- | ------------------- |
| `{"type": "ephemeral"}`              | 5 minutes | `cache_write_short` |
| `{"type": "ephemeral", "ttl": "1h"}` | 1 hour    | `cache_write_long`  |

Any `ttl` other than `"1h"` is five minutes. Use the short write for requests within five minutes of each other, such as one agent loop. Use the long write when requests are further apart. A long write pays back only if the prefix is read after the short TTL would have ended.

**Messages API** (Anthropic format). Mark a block of `system`, a tool, or a block inside a turn:

```python
import os
from anthropic import Anthropic

client = Anthropic(
    base_url="https://router.inference.ai.exchange",
    api_key=os.environ["LIQUID_API_KEY"],
)

response = client.messages.create(
    model="liquid.auto",
    max_tokens=1024,
    system=[
        {"type": "text", "text": "You are terse."},
        {
            "type": "text",
            "text": long_reference_document,
            "cache_control": {"type": "ephemeral", "ttl": "1h"},  # long write
        },
    ],
    messages=[{"role": "user", "content": "What changed in section 4?"}],
)
print(response.usage)
```

**Chat Completions.** Mark a content part or a tool. A marker needs the array form of `content`:

```python
response = client.chat.completions.create(
    model="liquid.auto",
    messages=[
        {
            "role": "system",
            "content": [
                {
                    "type": "text",
                    "text": long_reference_document,
                    "cache_control": {"type": "ephemeral"},  # short write
                }
            ],
        },
        {"role": "user", "content": "What changed in section 4?"},
    ],
)
```

Rules for markers:

- A request can have four breakpoints at most. The exchange drops the shallowest extra ones.
- Anthropic refuses a one-hour breakpoint after a five-minute one, so each breakpoint before a one-hour breakpoint also becomes one hour.
- A marker on a tool puts the breakpoint after the last tool.
- A top-level `cache_control` object puts one breakpoint after the last message, in both APIs. A value that is not an object places nothing:

  ```python
  response = client.chat.completions.create(
      model="liquid.auto",
      messages=messages,
      extra_body={"cache_control": {"type": "ephemeral"}},
  )
  ```

### Switch off the cache (`off`)

`off` drops your own markers and the cache key. It cannot stop an upstream that caches long prompts itself.

```python
response = client.chat.completions.create(
    model="liquid.auto",
    messages=[{"role": "user", "content": "hello"}],
    extra_body={"exchange": {"cache": {"mode": "off"}}},
)
```

## Keep a conversation on one cache

### Name the conversation

Send `session_id` in the request body, or the `x-session-id` header, with the same value on each turn. The body field wins. The value can have 256 characters at most.

The exchange picks the cache key in this order: `session_id`, `prompt_cache_key`, then a digest of the first block of the prompt. Some providers, such as OpenAI, route by this key. The exchange salts the key for each buyer, so it never joins the requests of two buyers. With `exchange.cache.mode: "off"`, the exchange ignores `session_id`.

```python
response = client.chat.completions.create(
    model="liquid.auto",
    messages=messages,
    extra_body={"session_id": "support-chat-4821"},
)
```

### Which provider serves the next turn

A cache lives at one provider. The exchange remembers which provider served the last turn of a session, for as long as the longest cache that the turn wrote. Each new turn renews the record.

For that provider, the ranking estimate assumes a cache read for part of the prompt, but only if the provider publishes a cache-read rate.

- The assumed share is the measured hit rate of the deployment over 30 days of settled jobs, or one half with too little history.
- To set the share yourself, use `exchange.cache.expected_hit_rate` in [Purchasing rules](https://inference.ai.exchange/docs/rules.md).

When the cache saves anything, the exchange also ranks that provider 5% below its own estimate. So the provider that served the last turn wins unless another offer is cheaper by more than the cache saving plus that margin. The price cap does not include the assumed saving. Under Auto, a session can also stay on the previous deployment. See [Auto Router](https://inference.ai.exchange/docs/auto.md).

A new conversation can also find a warm cache. The exchange remembers, for each buyer, which provider served each prefix, for the same time as a session. This works without a `session_id`, and the exchange can lose a record early.

### Keep a session on one deployment

Set `exchange.session.sticky: true` in the `exchange` block or in a preset. The deployment that served the previous turn then ranks first while it is eligible: it passes your rules and its price cap fits the cap of the call. Otherwise the exchange ranks as usual. A fallback to the next offer happens only before any output.

```python
response = client.chat.completions.create(
    model="liquid.auto",
    messages=messages,
    extra_body={"session_id": "support-chat-4821", "exchange": {"session": {"sticky": True}}},
)
```

`exchange.session.sticky` is not the Auto session pin of Auto versions 5 and 6. Set `exchange.session.pin: "off"` to turn the pin off, or `exchange.session.margin` to change the score points that a rival must add. See [Session pin](https://inference.ai.exchange/docs/auto.md).

## Privacy between buyers

Only you can read your cached prompts. The exchange puts a code that is unique to your account before your system prompt and messages in each cacheable prompt, so a shared upstream account never shares a prefix between buyers. One exception: some upstreams cache the tool definitions before the system prompt, so buyers can share the cache for an identical list of tools.

## Reference

### The meters

The exchange meters the input of a job in four parts and the output as a fifth. Each part has its own rate on the offer.

| Meter               | What it counts                               | Typical rate                     |
| ------------------- | -------------------------------------------- | -------------------------------- |
| `uncached_input`    | Input that the cache did not read or write.  | The input rate.                  |
| `cache_read`        | Input that the provider read from its cache. | 0.1 to 0.5 times the input rate. |
| `cache_write_short` | Input stored in the cache for five minutes.  | 1 to 1.25 times the input rate.  |
| `cache_write_long`  | Input stored in the cache for one hour.      | About 2 times the input rate.    |
| `generation`        | Output tokens. Caching has no effect.        | The output rate.                 |

The public offers in [Market data](https://inference.ai.exchange/docs/market.md) carry the rates in the `cache_read`, `cache_write_short` and `cache_write_long` lines of `pricing`. A line is absent when the offer quotes no such price.

### What each upstream caches

The exchange sends the same cache plan to each provider, in the form that its upstream reads. The upstream decides what it caches and what it charges. The multipliers below are list prices of the upstream, and an offer can quote other rates. A prompt shorter than the minimum of the upstream is never cached and is billed as uncached input.

#### Anthropic

- You choose where the cache starts, with `cache_control` markers or exchange breakpoints.
- A five-minute write costs 1.25 times the input rate, a one-hour write 2 times, and a read 0.1 times (less on some newer models).
- The minimum cacheable prompt depends on the model, from 512 to 4,096 tokens. A prompt below it is processed without a cache and with no error, also with a marker. See the Anthropic documentation for the minimum of each model.
- The cache checks at most 20 blocks back from each breakpoint. In a long turn, place a new breakpoint.
- A change to the tool definitions invalidates the tools, the system prompt and the messages. A change to the system prompt invalidates the system prompt and the messages.

#### OpenAI

- OpenAI caches a long prefix by itself. On GPT-5.6 and later, the minimum is 1,024 visible input tokens.
- On GPT-5.6 and later, a write costs 1.25 times the input rate and a read 0.1 times. Earlier models have no write charge.
- The exchange bills written tokens at the write rate of the offer only when the request has a cache breakpoint. Otherwise it bills them as uncached input.
- The exchange sends `prompt_cache_key`, and OpenAI routes requests with the same key to the same cache. Keep the start of the prompt the same and put changing text at the end.

#### Other upstreams

Groq, Grok, Moonshot AI, DeepSeek, Z.AI, Google Gemini and others cache a repeated prefix by themselves. The exchange sends the same body as for OpenAI, and the upstream can ignore the key.

- A read costs less than input. A write is usually free, and the exchange then bills written tokens as uncached input.
- DeepSeek reports cached input as `prompt_cache_hit_tokens`. The exchange bills these tokens at the cache-read rate of the offer.
- Alibaba caches only at `cache_control` markers, with a five-minute TTL, and charges for the write.
- The exchange does not create explicit Gemini caches. Put stable text first.

These rules are a guide. The `usage` block shows what the provider reported, and the receipt shows what you paid.

## Next steps

- [Response caching](https://inference.ai.exchange/docs/response-caching.md): store a whole answer and get it back at no charge.
- [Purchasing rules](https://inference.ai.exchange/docs/rules.md): the `exchange.cache.mode` setting and `exchange.session.sticky`.