Cold starts and what is billed

Endpoints sleep when nobody uses them, so you pay nothing at night or on quiet days. The price of sleeping is a short wait the next time someone calls: a cold start.

What one wake costs

1. AsleepFree
2. Starting a GPUFree
3. Loading the modelFree
4. Answering requestsBilled per second
5. Waiting before sleepBilled, you pick 60 s to 1 h

• Free: while asleep, while a GPU machine starts, while the model downloads and loads, and while you wait for a free GPU.
• Billed by the second: from the moment the model is ready until it goes back to sleep. That includes the wait after the last request (the Sleep after time), because the GPU stays on, ready to answer at once.

How long a cold start takes

Large models take longer, mostly to download. When every GPU of a size is busy, the start waits for a free one and keeps trying in every zone and region. If none is free for 30 minutes, it stops trying and tells you (no_capacity). You are not billed while it waits.

What your code sees

• Every request waits on the open connection for up to 8 minutes, then answers normally. You do not need to retry.
• While it waits, BitsRun sends something your client ignores, so no proxy closes the connection: keep-alive comments for streaming requests, spaces before the JSON for other requests (JSON parsers skip them), and empty progress lines for Ollama native streams.
• If the model is still not ready after 8 minutes (a very large model or a slow download), or it fails to start, the answer has status 200 and an error body with "code": "model_starting" or "model_unavailable". The wake carries on, so send the request again.
• Up to 4 requests wait at once per endpoint for your account. More than that get 503 model_starting with Retry-After: 30, so a retry loop cannot pile up answers nobody reads. Requests that time out on your side still run once the model is ready, and are billed.
• Prefer a 503? Send the header X-BitsRun-Wait: 0. A request that finds the model asleep then waits about 45 seconds and gets 503 with "code": "model_starting" and Retry-After: 30. Use it when your code retries on its own or must see real status codes.

Your own retry loop in Python, with the header:

import time
from openai import OpenAI, APIStatusError

# X-BitsRun-Wait: 0 answers 503 while the model starts, so this loop decides how long to wait.
client = OpenAI(base_url="https://<id>.endpoint.bitsrun.com/v1", api_key="brk_...",
                default_headers={"X-BitsRun-Wait": "0"}, max_retries=0)

def ask(**kwargs):
    for attempt in range(16):
        try:
            return client.chat.completions.create(**kwargs)
        except APIStatusError as e:
            if e.status_code != 503:
                raise
            time.sleep(30)
    raise RuntimeError("The endpoint did not wake in 20 minutes")

Avoiding the wait

• Select Start on the endpoint card a few minutes before you need it.
• Choose a longer Sleep after (up to 1 hour) for bursty traffic. You pay while it waits.
• Turn on Priority so a cold start never waits for a spare GPU.
• Turn on Always on for no cold starts at all.
• Tick "Notify me when it is ready" on the card, or turn on the ready email.

These are explained in Sleep, always-on and priority and Notifications.

A worked example

A support chat on an NVIDIA L4 ($0.000264 a second) with Sleep after set to 2 minutes gets a burst of questions that keeps it answering for 10 minutes.

• Cold start, 3 minutes: free.
• Answering, 10 minutes (600 seconds): $0.16.
• Waiting before sleep, 2 minutes (120 seconds): $0.03.
• Total for the burst: about $0.19.