Limits

Limits keep one busy app from slowing everyone down and protect your balance from runaway code. Most apps never reach them.

Requests open at once

This is how many requests your account can have in progress at the same moment, across all your endpoints. A request counts from when it arrives until its answer finishes.

A request over the limit gets 429 rate_limited right away. Wait a moment and retry, ideally with a short, growing pause (for example 1, 2, then 4 seconds). To raise the limit, move to a bigger plan; see Plans.

Each endpoint also has its own Requests at once setting: how many answers the model works on together. More at once means more total throughput but less memory for each conversation. Change it under Performance.

Request size and fields

• Replies are capped at 32,768 tokens (max_tokens, max_completion_tokens and Ollama's num_predict).
• n (several replies to one request) is capped at 4.
• Only the fields each path needs are passed to the model; others are dropped. The lists are on Chat and completions and Embeddings.
• How much a model can read at once (its context, or memory length) is set per endpoint, from 2,048 tokens up to what the GPU allows, at most 65,536. A longer request gets 400 context_too_long.

Time limits

• Starting from sleep: every request waits up to 8 minutes on the open connection. With the header X-BitsRun-Wait: 0, it gets 503 model_starting with Retry-After: 30 after about 45 seconds instead.
• Long answers: BitsRun keeps the connection alive while the model works, so slow answers are not cut off by a proxy. A streaming answer gets keep-alive comments; a non-streaming one gets spaces before the JSON, which JSON parsers ignore.
• Stopping early: closing the connection does not stop the model. It finishes the answer, up to max_tokens, and the answer is billed. Set max_tokens to limit what an abandoned request can cost.
• Waiting requests: each account can have 4 requests per endpoint waiting for it to start. More requests than that get 503 model_starting with Retry-After: 30 after about 45 seconds, and the start carries on.
• Waiting for a GPU: when every GPU of a size is busy, BitsRun keeps trying for up to 30 minutes without billing you. After that, requests get 503 no_capacity with a Retry-After header until it tries again.

Balance

An endpoint answers only while your balance is above $0. At $0 or below, running endpoints stop within seconds and every request gets 402 payment_required until you add credit. See Add credit and auto top-up.

Other account limits

An endpoint can change its settings 6 times an hour. Need more of something? Write to support@bitsrun.com.

What can go wrong

• 429 rate_limited: too many requests open at once. Retry with a pause, send fewer requests in parallel, or move to a bigger plan.
• 429 too_many_changes: an endpoint changed settings 6 times in the last hour. Wait, then try again.
• 409 endpoint_limit_reached: your plan has no room for another endpoint. Delete one you no longer use, or move to a bigger plan.