Streaming

Set stream: true and the answer arrives as server-sent events. The format is the standard chat.completion.chunk shape, so any OpenAI-compatible client parses it without changes.

1stream = client.chat.completions.create(
2    model="upfyn-yuva",
3    messages=[{"role": "user", "content": "Count to five."}],
4    stream=True,
5    stream_options={"include_usage": True},
6)
7 
8for chunk in stream:
9    if chunk.choices and chunk.choices[0].delta.content:
10        print(chunk.choices[0].delta.content, end="", flush=True)
11    elif chunk.usage:                      # the final chunk carries no choices
12        print(f"\n\ncost: {chunk.usage.total_tokens} tokens")

What the stream looks like#

1data: {"id":"chatcmpl-...","object":"chat.completion.chunk","model":"upfyn-yuva",
2       "choices":[{"index":0,"delta":{"role":"assistant","content":""}}]}
3 
4data: {"id":"chatcmpl-...","choices":[{"index":0,"delta":{"content":"One"}}]}
5 
6data: {"id":"chatcmpl-...","choices":[{"index":0,"delta":{"content":", two"}}]}
7 
8data: {"id":"chatcmpl-...","choices":[{"index":0,"delta":{},"finish_reason":"stop"}]}
9 
10data: {"id":"chatcmpl-...","choices":[],"usage":{"prompt_tokens":12,"completion_tokens":8,
11       "total_tokens":20},"credits_used":0.09,"credits_remaining":4821.35}
12 
13data: [DONE]

• Each frame is data: followed by one JSON object.
• Text arrives in choices[0].delta.content.
• finish_reason appears on the last content-bearing frame.
• The stream always ends with data: [DONE].

Getting the cost#

By default a stream tells you nothing about what it cost. Ask for it with stream_options: { include_usage: true } and one extra frame is sent before [DONE], carrying usage, credits_used and credits_remaining. That frame has an empty choices array.

Keepalive comments#

When a model takes a while to produce its first token, we flush headers early and send SSE comment frames so proxies and load balancers do not close the connection.

1: upfyn-thinking
2 
3: upfyn-thinking
4 
5data: {"id":"chatcmpl-...","choices":[{"index":0,"delta":{"content":"Here"}}]}

Lines starting with : are comments in the SSE specification, and every conformant parser — including both OpenAI SDKs — ignores them. You only need to care if you wrote your own parser: skip any line that does not begin with data: .

When a stream fails#

The status code is already 200 by the time the first byte leaves, so a mid-stream failure arrives as a frame with an error key, then [DONE]. Check every frame for it — see Errors.

Resuming a dropped stream#

If a client disconnects, the gateway keeps reading from the model to completion, so the turn finishes and bills correctly rather than being abandoned half-done. That also means you can reconnect and collect what you missed.