Concepts

Tool calling

Utmost supports OpenAI-style function tools on chat completions, function tools on Responses, and Anthropic tools on Messages. This page uses chat completions.

Check capabilities.tools in GET /v1/models. With tools attached, the router only picks models that support them. A pinned model is not checked; the request goes to it as sent.

Defining tools

{
  "model": "auto",
  "messages": [{ "role": "user", "content": "Is CVE-2024-3094 in xz 5.6.1?" }],
  "tools": [
    {
      "type": "function",
      "function": {
        "name": "lookup_cve",
        "description": "Look up a CVE and the affected versions.",
        "parameters": {
          "type": "object",
          "properties": { "id": { "type": "string" } },
          "required": ["id"]
        },
        "strict": true
      }
    }
  ]
}
  • type must be function.
  • name: 1 to 64 letters, digits, _ or -.
  • parameters: a JSON Schema object.
  • Up to 128 tools per request.

Tools count toward input tokens and toward the router's difficulty estimate (any tools add one point; see Routing).

tool_choice

Value Meaning
"auto" The model decides.
"none" No tool calls.
"required" The model must call at least one tool.
{"type": "function", "function": {"name": "lookup_cve"}} The model must call this tool.

parallel_tool_calls: false asks for at most one call per turn. Both settings are passed to the backend; how strictly they are honoured depends on the model.

The response

{
  "choices": [{
    "index": 0,
    "message": {
      "role": "assistant",
      "content": null,
      "tool_calls": [{
        "id": "call_a5fcd44539fe088d",
        "type": "function",
        "function": { "name": "lookup_cve", "arguments": "{\"id\":\"CVE-2024-3094\"}" }
      }]
    },
    "finish_reason": "tool_calls"
  }]
}

arguments is a JSON string. Parse it, and validate it before acting on it.

Sending results back

Append the assistant message with its tool_calls, then one tool message per call:

{
  "model": "auto",
  "tools": [ ... ],
  "messages": [
    { "role": "user", "content": "Is CVE-2024-3094 in xz 5.6.1?" },
    { "role": "assistant", "content": null, "tool_calls": [
      { "id": "call_a5fcd44539fe088d", "type": "function", "function": { "name": "lookup_cve", "arguments": "{\"id\":\"CVE-2024-3094\"}" } }
    ]},
    { "role": "tool", "tool_call_id": "call_a5fcd44539fe088d", "content": "{\"affected\": [\"5.6.0\", \"5.6.1\"]}" }
  ]
}

With an auto alias, each turn is routed on its own, so turns of one conversation can run on different models. Pin a model if that matters.

Parallel calls

A model may return several entries in tool_calls. Answer each with its own tool message, matching tool_call_id, in any order.

Streaming

Tool calls stream as deltas on choices[].delta.tool_calls. Each delta has an index. The first delta for an index carries id, type and function.name. Later deltas carry pieces of function.arguments to append.

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"id":"call_a5fc...","type":"function","function":{"name":"lookup_cve","arguments":""}}]},"finish_reason":null}], ...}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"{\"id\":\"CVE-"}}]},"finish_reason":null}], ...}

data: {"choices":[{"index":0,"delta":{"tool_calls":[{"index":0,"function":{"arguments":"2024-3094\"}"}}]},"finish_reason":null}], ...}

data: {"choices":[{"index":0,"delta":{},"finish_reason":"tool_calls"}], ...}

Assemble by index:

import OpenAI from "openai";

const client = new OpenAI({ apiKey: process.env.UTMOST_API_KEY, baseURL: "https://api.utmost.site/v1" });

const stream = await client.chat.completions.create({ model: "auto", stream: true, tools, messages });
const calls: { id: string; name: string; arguments: string }[] = [];
for await (const chunk of stream) {
  for (const d of chunk.choices[0]?.delta?.tool_calls ?? []) {
    const c = (calls[d.index] ??= { id: "", name: "", arguments: "" });
    if (d.id) c.id = d.id;
    if (d.function?.name) c.name = d.function.name;
    c.arguments += d.function?.arguments ?? "";
  }
}
console.log(calls.map((c) => ({ ...c, arguments: JSON.parse(c.arguments) })));
calls = {}
for chunk in client.chat.completions.create(model="auto", stream=True, tools=tools, messages=messages):
    if not chunk.choices:
        continue
    for d in chunk.choices[0].delta.tool_calls or []:
        c = calls.setdefault(d.index, {"id": "", "name": "", "arguments": ""})
        c["id"] = d.id or c["id"]
        if d.function and d.function.name:
            c["name"] = d.function.name
        c["arguments"] += (d.function.arguments or "") if d.function else ""

Notes

  • The refusal retry never runs on a response that contains tool calls.
  • Tool results are ordinary input. They are checked against the hard limits like any other content.