Concepts

Streaming

Set "stream": true on /v1/chat/completions, /v1/responses or /v1/messages. The response is text/event-stream.

What is different about streams

  • Headers come first. x-request-id, x-utmost-model, x-utmost-route, x-utmost-fallback and the rate-limit headers are sent. The token and cost headers are not, because nothing has been generated yet. Cost arrives in the final event.
  • No refusal retry. The router cannot retry a refusal it has already streamed to you. See Routing.
  • No output safety check. For abliterated models, the gateway checks the input of every request, but checks the output only on non-streamed responses. See Safety.
  • Fallback only before the first byte. If a backend fails before responding, the next one is tried. Once tokens are flowing, a failure ends the stream with an error event.
  • Disconnects are billed. If you close the connection early, you are billed for the tokens generated up to that point.

Chat completions

Each event is a data: line holding a chat.completion.chunk. The stream ends with data: [DONE].

data: {"id":"chatcmpl-req_...","object":"chat.completion.chunk","created":1795066905,"model":"utmost/qwen3-30b-a3b","choices":[{"index":0,"delta":{"role":"assistant","content":""},"finish_reason":null}]}

data: {"id":"chatcmpl-req_...","object":"chat.completion.chunk","created":1795066905,"model":"utmost/qwen3-30b-a3b","choices":[{"index":0,"delta":{"content":"Hello"},"finish_reason":null}]}

data: {"id":"chatcmpl-req_...","object":"chat.completion.chunk","created":1795066905,"model":"utmost/qwen3-30b-a3b","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}

data: [DONE]

Deltas can carry content, reasoning_content and tool_calls. See Tool calling for tool deltas.

The usage chunk

Send "stream_options": {"include_usage": true} to get one more chunk before [DONE]. It has empty choices and the final usage, including cost_usd:

{
  "id": "chatcmpl-req_...",
  "object": "chat.completion.chunk",
  "created": 1795066905,
  "model": "utmost/qwen3-30b-a3b",
  "choices": [],
  "usage": {
    "prompt_tokens": 5,
    "completion_tokens": 22,
    "total_tokens": 27,
    "prompt_tokens_details": { "cached_tokens": 0 },
    "completion_tokens_details": { "reasoning_tokens": 0 },
    "cost_usd": 0.000007
  }
}

Without include_usage, a stream tells you nothing about its cost. The request is still metered, and the cost appears in the dashboard. The Utmost SDKs set include_usage by default.

Responses

Each event has an event: line naming its type and a data: line whose JSON repeats the type and carries a sequence_number. There is no [DONE] sentinel; the stream ends after response.completed or response.incomplete.

event: response.created
data: {"type":"response.created","sequence_number":0,"response":{"id":"resp_...","status":"in_progress",...}}

event: response.output_text.delta
data: {"type":"response.output_text.delta","sequence_number":4,"item_id":"msg_...","output_index":0,"content_index":0,"delta":"Hello"}

event: response.completed
data: {"type":"response.completed","sequence_number":9,"response":{"id":"resp_...","status":"completed","usage":{...,"cost_usd":0.000007},...}}

The full list of event types is on the Responses page. Usage, with cost_usd, is in the response object of the last event.

Messages (Anthropic)

The Anthropic event sequence:

Event Notes
message_start usage here is all zeros
ping Once, after message_start
content_block_start A thinking, text or tool_use block
content_block_delta thinking_delta, text_delta or input_json_delta
content_block_stop
message_delta stop_reason and the final usage
message_stop

The Anthropic usage object has no cost field. Read cost from the dashboard, or use chat completions or Responses when you need it per request.

Errors mid-stream

An error before the stream starts is an ordinary JSON error response with an HTTP status. An error after the stream starts arrives as a final event, after which the server closes the connection. The HTTP status is already 200 by then.

Endpoint Error event
Chat completions data: {"error":{"message":"...","type":"api_error","code":"upstream_error","param":null,"request_id":"req_..."}}, with no [DONE] after it
Responses event: error with data: {"type":"error","error_type":"api_error","message":"...","code":"upstream_error","param":null,"request_id":"req_...","sequence_number":N}
Messages event: error with data: {"type":"error","error":{"type":"api_error","message":"...","code":"upstream_error"},"request_id":"req_..."}

The usual mid-stream code is upstream_error: the backend's stream ended unexpectedly. Retrying is safe, but you are billed for the tokens the failed stream generated.

In the Responses error event, type is always error; the error's own type (for example api_error) is in error_type.

Reading a stream without an SDK

import json, os, urllib.request

req = urllib.request.Request(
    "https://api.utmost.site/v1/chat/completions",
    data=json.dumps({
        "model": "auto", "stream": True, "stream_options": {"include_usage": True},
        "messages": [{"role": "user", "content": "Say hello"}],
    }).encode(),
    headers={"Authorization": f"Bearer {os.environ['UTMOST_API_KEY']}", "Content-Type": "application/json"},
)
with urllib.request.urlopen(req) as resp:
    for raw in resp:
        line = raw.decode().strip()
        if not line.startswith("data: "):
            continue
        data = line[6:]
        if data == "[DONE]":
            break
        chunk = json.loads(data)
        if "error" in chunk:
            raise RuntimeError(chunk["error"]["message"])
        for choice in chunk["choices"]:
            print(choice["delta"].get("content") or "", end="", flush=True)
        if chunk.get("usage"):
            print("\ncost:", chunk["usage"]["cost_usd"])