Sell with an Anthropic API

If your API already speaks the Anthropic Messages dialect, you can sell with no agent and no delivery endpoint. The exchange calls your API directly. To start, log in with dispatch_dialect set to anthropic.

When to use this

Use this path when your API streams a message in the Anthropic format.

Set it up

  1. Register the provider and its deployments. See Register deployments.
  2. Log in with dispatch_dialect set to anthropic.
  3. Quote as usual.

Log in

dispatch_url is the base URL of your API. Its scheme, host and port must match the registered declared_endpoint. dispatch_token is the key of your API. The exchange sends it as x-api-key and keeps it in memory for this session only.

{
  "type": "login",
  "slug": "acme",
  "api_key": "sk-liquid-...",
  "dispatch_url": "https://api.acme.example/v1",
  "dispatch_token": "your-anthropic-api-key",
  "dispatch_dialect": "anthropic",
  "capabilities": []
}

Open the session and quote

This script logs in, quotes each listed deployment and then sends a heartbeat every two seconds. The exchange closes a session that sends no frame for five seconds. See Keep the session alive. The script uses the Python websockets package. A real provider also quotes again before the quote expires, and when its price changes.

import asyncio
import json
import websockets

URL = "wss://router.inference.ai.exchange/v1/provider/ws"

LOGIN = {
    "type": "login",
    "slug": "acme",
    "api_key": "sk-liquid-...",
    "dispatch_url": "https://api.acme.example/v1",
    "dispatch_token": "your-anthropic-api-key",
    "dispatch_dialect": "anthropic",
    "capabilities": [],
}

def quote(model_id: str, revision: int) -> dict:
    return {
        "type": "quote",
        "target": {"kind": "registered", "model_id": model_id},
        "pricing": [
          {"billable": "input_text", "unit": "mtok", "usd": 0.12},
          {"billable": "output_text", "unit": "mtok", "usd": 0.48}
        ],
        "service_class": {"ttft_ms": 1500, "completion_ms": 30000},
        "supported_features": ["text"],
        "retention_policy_supported": ["none"],
        "revision": revision,
    }

async def beat(socket) -> None:
    while True:
        await asyncio.sleep(2)
        await socket.send(json.dumps({"type": "heartbeat"}))

async def main() -> None:
    async with websockets.connect(URL) as socket:
        await socket.send(json.dumps(LOGIN))
        beating = asyncio.create_task(beat(socket))
        try:
            async for raw in socket:
                frame = json.loads(raw)
                print(frame["type"])
                if frame["type"] == "listed":
                    for model in frame["models"]:
                        await socket.send(json.dumps(quote(model["model_id"], 1)))
                elif frame["type"] in ("login_reject", "quote_reject"):
                    raise SystemExit(frame)
        finally:
            beating.cancel()

asyncio.run(main())

What the exchange sends

The exchange posts a streamed message to {dispatch_url}/messages. The header Idempotency-Key is the award id. Your server may ignore it. The exchange never sends an award twice after the request leaves.

curl https://api.acme.example/v1/messages \
  -H "x-api-key: your-anthropic-api-key" \
  -H "anthropic-version: 2023-06-01" \
  -H "Idempotency-Key: aw_8f3c…" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "acme-claude-large",
    "max_tokens": 64,
    "stream": true,
    "messages": [{ "role": "user", "content": "Name a prime number." }]
  }'

Stream server-sent events in the Messages format, and end with message_stop. The exchange settles on the counts in message_start.message.usage and message_delta.usage. The last value of each count wins.

event: message_start
data: {"type":"message_start","message":{"id":"msg_1","role":"assistant","usage":{"input_tokens":14,"cache_read_input_tokens":0,"cache_creation_input_tokens":0,"output_tokens":1}}}

event: content_block_start
data: {"type":"content_block_start","index":0,"content_block":{"type":"text","text":""}}

event: content_block_delta
data: {"type":"content_block_delta","index":0,"delta":{"type":"text_delta","text":"Seven is prime."}}

event: content_block_stop
data: {"type":"content_block_stop","index":0}

event: message_delta
data: {"type":"message_delta","delta":{"stop_reason":"end_turn"},"usage":{"output_tokens":5}}

event: message_stop
data: {"type":"message_stop"}

A stream with no usage counts, a stream that ends before message_stop, and an error event do not settle as a success.

Refuse a job

Answer an error with a status code of 400 or more before the first event, and the exchange offers the job to the next offer.

HTTP/1.1 529
Content-Type: application/json

{ "type": "error", "error": { "type": "overloaded_error", "message": "the deployment is at capacity" } }

After the first event of a chat answer, the exchange cannot fall back to another offer, because the buyer would get one answer from two providers.

Limits