Chat and completions

Chat is the main way to talk to a language model. You send the conversation so far as a list of messages, and the model writes the next one.

Send a chat request

Each message has a role and content. Roles are system (instructions for the model), user (the person) and assistant (earlier replies from the model).

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://01k8y4000d3m00000000000011.endpoint.bitsrun.com/v1",
    api_key=os.environ["BITSRUN_API_KEY"],
)
reply = client.chat.completions.create(
    model="qwen3:8b",
    messages=[
        {"role": "system", "content": "You answer support questions for an online bike shop. Be brief."},
        {"role": "user", "content": "How do I return a helmet?"},
    ],
    max_tokens=300,
    temperature=0.3,
)
print(reply.choices[0].message.content)

The model does not remember earlier calls. To continue a conversation, send all earlier messages again, including the assistant's replies.

Stream the answer

Streaming sends the reply in small pieces as the model writes it, so people see text right away. While a sleeping endpoint starts, any request waits on the open connection; streaming shows the first words as soon as it is ready.

stream = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "Write a haiku about trains."}],
    stream=True,
)
for chunk in stream:
    if chunk.choices and chunk.choices[0].delta.content:
        print(chunk.choices[0].delta.content, end="", flush=True)

const stream = await client.chat.completions.create({
  model: "qwen3:8b",
  messages: [{ role: "user", content: "Write a haiku about trains." }],
  stream: true,
});
for await (const chunk of stream) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}

With curl, add "stream": true and -N so curl prints each piece as it arrives:

curl -N https://01k8y4000d3m00000000000011.endpoint.bitsrun.com/v1/chat/completions \
  -H "Authorization: Bearer $BITSRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3:8b", "stream": true, "messages": [{"role": "user", "content": "Count to five."}]}'

Closing the connection does not stop the model. It finishes the answer, up to max_tokens, and that answer is billed like any other. To keep an answer you might abandon cheap, set max_tokens to what you need.

Fields you can send

BitsRun passes these fields to the model and drops any others:

• messages, stream, stream_options
• max_tokens or max_completion_tokens (the longest reply, capped at 32,768 tokens)
• temperature, top_p, seed, stop, presence_penalty, frequency_penalty
• n (number of replies, up to 4), logprobs, top_logprobs
• tools, tool_choice, parallel_tool_calls (see Tool calling)
• response_format (for JSON output)
• chat_template_kwargs with enable_thinking only (see below)

A token is a piece of a word, about four characters of English text on average.

Turn thinking on or off

Some models, such as Qwen3, think out loud before they answer. Thinking gives better answers to hard questions but takes longer. Where the model supports it, switch it off for quick replies:

reply = client.chat.completions.create(
    model="qwen3:8b",
    messages=[{"role": "user", "content": "What is the capital of Peru?"}],
    extra_body={"chat_template_kwargs": {"enable_thinking": False}},
)

Plain completions

/v1/completions continues a piece of text instead of a conversation. It accepts prompt, suffix, echo and the same sampling fields. Use chat unless you know you need this.

curl https://01k8y4000d3m00000000000011.endpoint.bitsrun.com/v1/completions \
  -H "Authorization: Bearer $BITSRUN_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model": "qwen3:8b", "prompt": "The three primary colors are", "max_tokens": 30}'

LangChain

LangChain's OpenAI chat model works with the same two settings:

import os
from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    base_url="https://01k8y4000d3m00000000000011.endpoint.bitsrun.com/v1",
    api_key=os.environ["BITSRUN_API_KEY"],
    model="qwen3:8b",
)
print(llm.invoke("Give me one tip for writing clear emails.").content)

Ollama clients

GGUF models (deployed from an Ollama name or a .gguf file, served by llama.cpp) also speak Ollama's own API. Models served by vLLM or TEI answer only the OpenAI paths. Use the endpoint host without /v1 and send your key in the Authorization header. The Ollama Python library takes it in headers:

import os
from ollama import Client

client = Client(
    host="https://01k8y4000d3m00000000000011.endpoint.bitsrun.com",
    headers={"Authorization": f"Bearer {os.environ['BITSRUN_API_KEY']}"},
)
reply = client.chat(model="qwen3:8b", messages=[{"role": "user", "content": "Hello"}])
print(reply.message.content)

The ollama command line app (ollama run) cannot send that header, so it gets a 401. Use a library or call the API directly.

Or call it directly:

curl https://01k8y4000d3m00000000000011.endpoint.bitsrun.com/api/chat \
  -H "Authorization: Bearer $BITSRUN_API_KEY" \
  -d '{"model": "qwen3:8b", "messages": [{"role": "user", "content": "Hello"}], "stream": false}'

Only the chat, generate, embed and tags paths answer. Commands that change the server, such as pull, create or delete, are not available.

What can go wrong

• model_starting: the endpoint was asleep and took longer than 8 minutes to start (in a 200 body), or you sent X-BitsRun-Wait: 0 (a 503). Send the request again.
• 400 context_too_long: the conversation is longer than the model can read at once. Drop older messages, or raise Memory length under Performance.
• 401 unauthorized: the key is missing, mistyped or revoked. Check the Authorization: Bearer header.
• 429 rate_limited: your account has too many requests open at once. See Limits.

Every code is explained on Error codes.