Streaming Tokens to the User: SSE, WebSockets, and the Wait the Reader Feels

Decoding turned a model’s distributions into tokens, one forward pass each. A 300-word answer is about 400 tokens, and at 20 to 50 milliseconds a token that’s eight to twenty seconds of generation. Nothing about decoding can change that the tokens come out one at a time. What can change is when the reader sees them.

Streaming sends each piece of text as soon as it exists, instead of the whole answer at the end. It doesn’t make generation faster: the last token arrives when it would have anyway. It changes which wait the reader feels. This post builds a streaming server from the standard library, measures what it buys, and goes through what breaks it in practice.

Table of contents

Five ways to get text to a browser

The web was built for request and response: the client asks, the server answers, the connection is done. Generated text doesn’t fit that shape, since the answer exists only gradually. There are five standard ways to deal with it:

Nearly every model API streams with SSE: OpenAI’s, Anthropic’s and Google’s all do. It’s plain HTTP, so it passes through the same infrastructure as everything else, and generation only needs the server-to-client direction. WebSockets come in when the client must also talk during the stream, as voice interfaces do (see System Design for AI); OpenAI’s Realtime API, for voice, uses WebSockets or WebRTC.

The SSE wire format

An SSE response is an ordinary HTTP response with the content type text/event-stream, kept open, into which the server writes events as lines of text (WHATWG, HTML Living Standard):

• data: ... lines carry the payload; several data lines in one event are joined with newlines;
• event: ... names the event’s type, id: ... gives it an identifier, and retry: ... tells the client how long to wait before reconnecting;
• a blank line ends the event and dispatches it;
• a line starting with : is a comment, ignored, which makes it the way to send keep-alive pings through proxies that close idle connections.

If the connection drops, the browser’s EventSource reconnects by itself and sends the last id it saw in a Last-Event-ID header, so the server can resume. But EventSource can only make GET requests, with no request body and no custom headers, and a model API needs to POST a prompt and an API key (which belongs on the application’s backend, never in the browser). So clients for model APIs read the response stream with fetch and parse the events themselves, and those conveniences don’t apply. They’d need help anyway: EventSource reconnects after a stream ends normally too, unless the server answers with status 204, and resuming at Last-Event-ID only works if the server kept what it already generated. Otherwise a reconnect starts generation over, and bills it again.

How the stream says it’s finished is a convention, not part of SSE:

• OpenAI’s Chat Completions end with a last event whose data is the literal [DONE]; its Python SDK stops reading when an event’s data starts with [DONE]. OpenAI’s newer Responses API uses named, typed events instead (response.output_text.delta, …, response.completed).
• Anthropic’s Messages API names every event: message_start, then for each content block content_block_start, a series of content_block_delta, and content_block_stop, then one or more message_delta and a final message_stop, with ping events anywhere and an error event if something fails mid-stream. New event types may be added, so a client should ignore ones it doesn’t know.

The server below uses the OpenAI-style sentinel, with one JSON object per piece of text.

A streaming server, from the standard library

The server, server.py, runs on asyncio with no web framework, with choose from Decoding for sampling and Qwen2.5-1.5B-Instruct as the model. It answers three ways: POST /complete returns the whole answer as one JSON response at the end, POST /stream sends server-sent events, and GET /ws opens a WebSocket. Generation runs in a thread and hands each piece of text to the network side through a queue; before every token it checks a stop flag:

def generate(prompt, n, out, stop, loop):
    """Sample up to n tokens (top-p 0.9, seeded), pushing text pieces to the
    asyncio queue `out`; a None marks the end."""
    m, ids = model(), torch.tensor([tok(prompt).input_ids], device=DEVICE)
    g = torch.Generator(device=DEVICE).manual_seed(0)
    d, history, produced = Detokenizer(), [], 0
    with torch.no_grad():
        result = m(ids, use_cache=True)
        for _ in range(n):
            if stop.is_set():
                break
            token = choose(result.logits[0, -1], history, g, p=0.9)
            history.append(token); produced += 1
            if token == tok.eos_token_id:
                break
            loop.call_soon_threadsafe(out.put_nowait, d.push(token))
            result = m(torch.tensor([[token]], device=DEVICE), past_key_values=result.past_key_values, use_cache=True)
    loop.call_soon_threadsafe(out.put_nowait, None)
    return produced

Full script: server.py

The request handler answers /complete with the whole text at once. Its SSE branch writes each piece as an event, and a second task watches the connection for the client leaving:

    if path == "/complete":
        pieces = []
        while (piece := await out.get()) is not None:
            pieces.append(piece)
        payload = json.dumps({"text": "".join(pieces)}).encode()
        writer.write(b"HTTP/1.1 200 OK\r\nContent-Type: application/json\r\n"
                     b"Content-Length: %d\r\nConnection: close\r\n\r\n" % len(payload) + payload)
    else:
        writer.write(b"HTTP/1.1 200 OK\r\nContent-Type: text/event-stream\r\nCache-Control: no-cache\r\n"
                     b"X-Accel-Buffering: no\r\nConnection: close\r\n\r\n")
        if not NAIVE:                                     # the client closing its side means it's gone
            asyncio.create_task(watch(reader, stop))
        while (piece := await out.get()) is not None:
            if piece:
                writer.write(b"data: " + json.dumps({"text": piece}).encode() + b"\n\n")
                try:
                    await writer.drain()
                except ConnectionError:
                    left = True
                    if not NAIVE:
                        stop.set()
        writer.write(b"data: [DONE]\n\n")
        left = left or stop.is_set()
    LOG.append((path, await job, left))
    print(f"{path}: generated {LOG[-1][1]} tokens, client left early: {left}", flush=True)
    try:
        await writer.drain(); writer.close()
    except ConnectionError:
        pass

async def watch(reader, stop):
    await reader.read()                                   # returns b"" when the client disconnects
    stop.set()

Full script: server.py

The SSE response has neither a Content-Length nor chunked encoding, so its end is the connection closing. That’s legal HTTP/1.1, but it means a dropped connection looks like a finished answer at the HTTP level, which is one thing the [DONE] event is for: it tells the client the end was meant. Frameworks send chunked encoding instead, and HTTP/2 and HTTP/3 carry the body in frames. This is a teaching server, not a production one: it parses headers loosely and handles few errors.

The WebSocket part is the protocol written out (RFC 6455). The client asks to upgrade with a random key; the server proves it understood by hashing the key with a fixed string from the RFC and sending the hash back, and from then on both sides exchange frames: a byte with the frame’s type and a “final fragment” bit, the payload’s length in one, three or nine bytes, and the payload. Frames from the client must be masked: XORed with four random bytes the browser chooses and the page’s script can’t predict. Without that, a script could send bytes that an old proxy which doesn’t understand WebSockets reads as a fresh HTTP request, and the proxy would then cache the attacker’s reply as the response for some other site’s URL (RFC 6455, §10.3). The upgrade, the stream with its “stop” listener, and writing a frame and reading a masked one:

async def websocket(reader, writer, headers):
    accept = base64.b64encode(hashlib.sha1((headers["sec-websocket-key"] +
                              "258EAFA5-E914-47DA-95CA-C5AB0DC85B11").encode()).digest())
    writer.write(b"HTTP/1.1 101 Switching Protocols\r\nUpgrade: websocket\r\nConnection: Upgrade\r\n"
                 b"Sec-WebSocket-Accept: " + accept + b"\r\n\r\n")
    request = json.loads(await read_frame(reader))
    out, stop, job = await start(request["prompt"], request.get("max_tokens", 100))
    async def listen():                                   # the client can talk while the server streams
        while True:
            message = await read_frame(reader)
            if message is None or json.loads(message).get("stop"):
                stop.set(); return
    listener = asyncio.create_task(listen())
    while (piece := await out.get()) is not None:
        if piece:
            writer.write(frame(json.dumps({"text": piece}).encode()))
    writer.write(frame(json.dumps({"done": True}).encode()))
    await writer.drain()
    LOG.append(("/ws", await job, stop.is_set()))
    print(f"/ws: generated {LOG[-1][1]} tokens, client said stop: {stop.is_set()}", flush=True)
    listener.cancel(); writer.close()

def frame(payload, opcode=0x1):
    """A server-to-client frame: FIN + opcode, then the length, then the data."""
    n = len(payload)
    head = bytes([0x80 | opcode]) + (bytes([n]) if n < 126 else bytes([126]) + n.to_bytes(2, "big"))
    return head + payload

async def read_frame(reader):
    """A client-to-server frame; clients must mask their data with 4 random bytes."""
    try:
        b0, b1 = await reader.readexactly(2)
    except asyncio.IncompleteReadError:
        return None
    n = b1 & 0x7F
    if n == 126:
        n = int.from_bytes(await reader.readexactly(2), "big")
    mask, data = await reader.readexactly(4), await reader.readexactly(n)
    if b0 & 0x0F == 0x8:                                  # a close frame
        return None
    return bytes(b ^ mask[i % 4] for i, b in enumerate(data)).decode()

Full script: server.py

This server leaves out what a real WebSocket server needs. It doesn’t answer pings (opcode 9) with pongs, and since it treats every frame as text, a client library that sends a ping, as Python’s websockets does every 20 seconds by default, would break its “stop” listener. It doesn’t send its own close frame at the end, so a browser reports an unclean close. It has no 64-bit lengths, no fragmented messages, and no check of the Origin header, which matters because browsers don’t apply CORS to WebSockets and do send cookies: without that check, any page can open a socket to it as the user.

A trap in sending text one token at a time

The server doesn’t send tok.decode([token]) for each token. A token is a piece of a byte sequence, and one character can span several tokens:

from lm import tok

# Streaming text one token at a time has a trap: a token is a piece of a
# byte sequence, not of a string, and one character can span two tokens.
# Decoding each token alone then prints broken characters. The fix is
# incremental: decode everything so far, and emit only the new text once it
# no longer ends in a half-finished character.
class Detokenizer:
    def __init__(self):
        self.ids, self.sent = [], ""
    def push(self, token_id):
        """Add a token; return the new text that is safe to send (may be '')."""
        self.ids.append(token_id)
        text = tok.decode(self.ids)
        if text.endswith("�"):            # the last character isn't complete yet
            return ""
        new, self.sent = text[len(self.sent):], text
        return new

Full script: detokenizer.py

from lm import tok
from detokenizer import Detokenizer

# A character split across tokens, decoded one token at a time and
# incrementally.
text = "Con vẹt 🦜 nói: mô hình sinh văn bản từng token một 🫠"
ids = tok(text).input_ids
naive = [tok.decode([i]) for i in ids]
print(f"{len(ids)} tokens; decoded one by one:")
print("  " + " | ".join(naive))

d = Detokenizer()
pieces = [d.push(i) for i in ids]
print("incremental:")
print("  " + " | ".join(pieces))
print(f"broken pieces one by one: {sum('�' in p for p in naive)}, incremental: "
      f"{sum('�' in p for p in pieces)}; joined text equal to the original: {''.join(pieces) == text}")

Full script: split_chars.py

20 tokens; decoded one by one:
  Con |  v | ẹ | t |  � | � | � |  nói | : |  mô |  hình |  sinh |  văn |  bản |  từng |  token |  một |  � | � | �
incremental:
  Con |  v | ẹ | t |  |  |  🦜 |  nói | : |  mô |  hình |  sinh |  văn |  bản |  từng |  token |  một |  |  |  🫠
broken pieces one by one: 6, incremental: 0; joined text equal to the original: True

The parrot and the melting face are each split across three tokens, and decoding them one at a time produces replacement characters � that no later token can repair, since each piece was already sent. The incremental version holds a piece back until its last character is complete. The Vietnamese words in this sentence happen to split only between whole characters (“v”, “ẹ”, “t”), but emoji, rarer scripts and unusual accented letters hit this constantly with a byte-level tokenizer. Three caveats. Decoding the whole text every step costs time proportional to its length; production detokenizers decode only a short window of recent tokens (vLLM’s offsets, Hugging Face’s DecodeStream). This one never flushes: whatever it holds back when generation ends must be sent then. And it assumes decoding a prefix of the tokens gives a prefix of the text, which holds for byte-level BPE like Qwen’s but not for every tokenizer.

Measured: the first text, and the bytes

A client written on raw sockets, so it can time every byte and count it:

async def post(path, prompt, n, stop_after=None):
    """POST a prompt; returns (seconds to first text, seconds to the end,
    bytes received, pieces of text). Closes early after `stop_after` events."""
    r, w = await asyncio.open_connection("127.0.0.1", PORT)
    body = json.dumps({"prompt": prompt, "max_tokens": n}).encode()
    t0 = time.perf_counter()
    w.write(b"POST %s HTTP/1.1\r\nHost: x\r\nContent-Length: %d\r\n\r\n" % (path.encode(), len(body)) + body)
    first, received, pieces, buffer = None, 0, [], b""
    head_done = False
    while chunk := await r.read(65536):
        received += len(chunk); buffer += chunk
        if not head_done and b"\r\n\r\n" in buffer:
            head_done, buffer = True, buffer.split(b"\r\n\r\n", 1)[1]
        if head_done and path == "/stream":
            while b"\n\n" in buffer:
                event, buffer = buffer.split(b"\n\n", 1)
                data = event[len(b"data: "):]
                if data != b"[DONE]":
                    first = first or time.perf_counter() - t0
                    pieces.append(json.loads(data)["text"])
            if stop_after and len(pieces) >= stop_after:
                w.close(); await asyncio.sleep(1.0)       # leave, then give the server a moment to notice
                return first, time.perf_counter() - t0, received, pieces
    if path == "/complete":
        first = time.perf_counter() - t0
        pieces = [json.loads(buffer)["text"]]
    return first, time.perf_counter() - t0, received, pieces

async def main():
    rows = {"/complete": [], "/stream": [], "/ws": []}
    for prompt in prompts:
        rows["/complete"].append(await post("/complete", prompt, 100))
        rows["/stream"].append(await post("/stream", prompt, 100))
        rows["/ws"].append(await websocket(prompt, 100))
    for name, rs in rows.items():
        first = sum(r[0] for r in rs) / len(rs); total = sum(r[1] for r in rs) / len(rs)
        text = ["".join(r[3]) for r in rs]
        received = sum(r[2] for r in rs); chars = sum(len(t.encode()) for t in text)
        print(f"{name:10s} first text after {first:5.2f} s, all of it after {total:5.2f} s; "
              f"{received / 5:6.0f} bytes on the wire per answer, for {chars / 5:4.0f} bytes of text "
              f"({received / chars:.1f}x)")
    same = all("".join(a[3]) == "".join(b[3]) == "".join(c[3]) for a, b, c in zip(*rows.values()))
    print(f"the same text all three ways: {same}")

Five of the prompts from Decoding, 100 tokens each, delivered all three ways, on an Apple M4 laptop (all of this post’s code runs in about 2 minutes there, once the model is downloaded):

/complete  first text after  4.68 s, all of it after  4.68 s;    596 bytes on the wire per answer, for  488 bytes of text (1.2x)
/stream    first text after  0.13 s, all of it after  4.75 s;   2626 bytes on the wire per answer, for  488 bytes of text (5.4x)
/ws        first text after  0.13 s, all of it after  5.12 s;   2038 bytes on the wire per answer, for  488 bytes of text (4.2x)
the same text all three ways: True

The same text arrives all three ways, and it finishes at about the same time, 4.7 to 5.1 seconds; one run of five prompts is too small to rank the three on that. What differs is the first text: 4.68 seconds without streaming, about 0.1 to 0.2 seconds with either kind of streaming (0.13 on average here). That is the time to first token (TTFT), measured from sending the request: the prompt’s forward pass plus one sampling step. These prompts are short, the first 200 characters of a paragraph each. With a 10,000-token prompt, or a queue of other requests ahead, TTFT grows, and that is where Serving Metrics and Prefill–Decode Disaggregation picks it up.

Streaming costs bytes. Each piece of text, about 5 bytes, travels inside an event: data: {"text": "..."} and a blank line. For this server that’s 5.4 times the text’s own size, 4.2 with WebSocket frames, against 1.2 for one JSON response. Real APIs carry more per event:

import json

# What one streamed piece of text, "Two", costs on the wire in different
# formats, as compact JSON. The Anthropic event is the documented example's
# shape; the OpenAI chunk has the fields of the example in OpenAI's cookbook
# (current chunks also carry a padding field, below).
text = "Two"
ours = b"data: " + json.dumps({"text": text}).encode() + b"\n\n"
anthropic = (b"event: content_block_delta\ndata: " + json.dumps(
    {"type": "content_block_delta", "index": 0, "delta": {"type": "text_delta", "text": text}},
    separators=(",", ":")).encode() + b"\n\n")
openai = b"data: " + json.dumps(
    {"id": "chatcmpl-9lMgfRSWPHcw51s6wxKT1YEO2CKpd", "object": "chat.completion.chunk", "created": 1721075653,
     "model": "gpt-july-test", "system_fingerprint": "fp_e9b8ed65d2",
     "choices": [{"index": 0, "delta": {"content": text}, "logprobs": None, "finish_reason": None}]},
    separators=(",", ":")).encode() + b"\n\n"
payload = json.dumps({"text": text}).encode()
websocket = bytes([0x81, len(payload)]) + payload            # a server-to-client frame: 2 bytes of header
for name, b in (("this post's SSE event", ours), ("Anthropic content_block_delta", anthropic),
                ("OpenAI chat.completion.chunk", openai), ("this post's WebSocket frame", websocket)):
    print(f"{name:31s} {len(b):4d} bytes for {len(text.encode())} bytes of text")

Full script: event_bytes.py

this post's SSE event             23 bytes for 3 bytes of text
Anthropic content_block_delta    118 bytes for 3 bytes of text
OpenAI chat.completion.chunk     257 bytes for 3 bytes of text
this post's WebSocket frame       17 bytes for 3 bytes of text

An OpenAI chunk repeats the response’s identifier, model name and timestamp in every event: about 257 bytes to carry three bytes of text. Current chunks are larger still, and vary: OpenAI now pads each one with a random-length obfuscation field by default, because otherwise the size of each encrypted event reveals each token’s length to anyone watching the network, enough to reconstruct the opening of 29% of responses closely and infer the topic of 55% (Weiss et al., 2024). That sounds wasteful and mostly isn’t: at 20 to 100 events a second it’s tens of kilobytes a second, nothing next to what a web page loads. Bytes start to matter for many concurrent streams through one proxy, and for compression, below.

What the reader feels

Streaming helps because people read slower than models write:

import json
from lm import tok

# Is generation faster than reading? Tokens per word on the blog's own
# prose (the 20 real continuations from "Decoding"), against a reader's speed: 238
# words a minute for non-fiction, the average across studies (Brysbaert,
# 2019). Generation speed is from measure.py: 100 tokens in about 4.7 s.
texts = [p["human"] for p in json.load(open("prompts.json"))]
tokens = sum(len(tok(t).input_ids) for t in texts)
words = sum(len(t.split()) for t in texts)
per_word = tokens / words
reading = 238 / 60 * per_word                              # tokens a second a reader gets through
generating = 100 / 4.7
print(f"{per_word:.2f} tokens per word; a reader at 238 words a minute takes in {reading:.1f} tokens a second; "
      f"this laptop generates {generating:.0f}, {generating / reading:.1f} times as fast")
for n_words in (50, 300):
    n = n_words * per_word
    print(f"a {n_words}-word answer ({n:.0f} tokens): {n / generating:.1f} s to generate, {n_words / 238 * 60:.0f} s to read; "
          f"without streaming the reader waits {n / generating:.1f} s before the first word, with it about 0.13 s")

Full script: reading_speed.py

1.38 tokens per word; a reader at 238 words a minute takes in 5.5 tokens a second; this laptop generates 21, 3.9 times as fast
a 50-word answer (69 tokens): 3.2 s to generate, 13 s to read; without streaming the reader waits 3.2 s before the first word, with it about 0.13 s
a 300-word answer (415 tokens): 19.5 s to generate, 76 s to read; without streaming the reader waits 19.5 s before the first word, with it about 0.13 s

A reader takes in about 5.5 tokens a second; even a laptop generates four times that. So once the first piece arrives, the reader is never waiting: the text runs ahead of them. Without streaming, a 300-word answer means 20 seconds of nothing. With it, 0.13 seconds, then reading. The perceived latency is the time to first token, not the total, as long as generation stays ahead of reading. That second condition is the one serving systems have to protect under load: the time per output token, TPOT, from Serving Metrics and Prefill–Decode Disaggregation is the number that must stay below about 180 ms (one token per $1/5.5$ seconds) for a reader never to catch up. That’s an average, and averages hide stalls: a pause while the server runs someone else’s long prompt is what a reader notices, so Serving Metrics and Prefill–Decode Disaggregation looks at the slowest gaps between tokens, not only the mean. (And 238 words a minute is for English prose; the tokens per word differ for code and for other languages, and readers skim.)

When the user leaves

A user closes the tab halfway through an answer. What should the server do? Stop: every further token costs a forward pass, and the request’s KV cache, which can be gigabytes for a long context (see The KV Cache), stays allocated until generation ends. Two runs of the same experiment, one against the server above, which watches its connection, and one against the same server started with --naive, which doesn’t. The script reads how many tokens the server generated from its GET /log endpoint, which returns the server’s record of each request:

async def main(which):
    if which == "leave":
        await post("/stream", PROMPT, 100, stop_after=10)
        await asyncio.sleep(5)                            # let a server that didn't notice finish
        path, generated, left = (await server_log())[-1]
        print(f"SSE client left after 10 pieces: the server generated {generated} tokens in all")
    if which == "stop":
        await websocket(PROMPT, 100, stop_after=10)
        path, generated, stopped = (await server_log())[-1]
        print(f"WebSocket client said stop after 10 pieces: the server generated {generated} tokens in all")
    if which == "proxy":
        client.PORT = 8766
        first, total, received, pieces = await post("/stream", PROMPT, 100)
        print(f"through the proxy: first text after {first:.2f} s, the end after {total:.2f} s, "
              f"{len(pieces)} pieces, {received:,} bytes")

Watching the connection:

SSE client left after 10 pieces: the server generated 10 tokens in all

Started with --naive:

SSE client left after 10 pieces: the server generated 100 tokens in all

The server that notices stops at 10 tokens; the naive one generates all 100, 90 of them for nobody. (The 11th forward pass was already running when the client left, and its token was discarded: a check per token stops within one token, and a batched serving engine checks once per step.)

How the server notices matters, and no signal works everywhere:

• Writing: after the client closes, the operating system accepts the next write and the one after fails, so on a direct connection a server that only watches for write errors stops about one token late.
• Reading, as here: the read returns end-of-file the moment the client closes, so the stop is immediate. But end-of-file isn’t the same as gone: a client that half-closes its sending side after the request would look like it left.
• Behind a proxy, the server sees the proxy’s connection, not the user’s. nginx closes the upstream connection when its client aborts, by default; other hops vary.
• Over HTTP/2 or HTTP/3, cancelling one request resets one stream on a connection that stays open, so no end-of-file ever arrives; the server must watch for the stream reset.
• A client whose network just vanishes, a phone going to sleep, sends nothing at all, and neither signal fires until TCP gives up, minutes later.

Frameworks expose the transport’s own signal (Starlette’s request.is_disconnected(), or the disconnect message that cancels a StreamingResponse), serving engines abort the request on it, and a write timeout or a heartbeat catches the rest. The same check belongs on the non-streaming endpoint: a closed tab wastes a /complete request just as much, and this server doesn’t check there at all.

With a WebSocket the client doesn’t have to leave to stop. It sends a message on the same connection:

WebSocket client said stop after 10 pieces: the server generated 10 tokens in all

With SSE, which only goes one way, “stop” means closing the connection, or a second request to a cancel endpoint.

Backpressure is the other direction of the same problem: the client is there but slow, a phone on a poor network. Here the generation thread puts pieces into an unbounded queue, so a slow reader makes the queue grow while the model runs at full speed. A bounded queue makes the generator wait when the reader falls behind; whether that’s wanted depends on whether the GPU time is better spent on other users. Text is small enough that most servers buffer it and move on, but it’s a decision, not a default.

The proxy that undoes it all

Between a user and a model server there are usually several machines: a CDN, a load balancer, a reverse proxy such as nginx. Proxies can buffer responses, collecting upstream data before passing it on, which is efficient for ordinary pages and fatal for a stream. Whether a given hop does depends on the product and its configuration. nginx’s proxy_buffering, on by default, stores the response in buffers of one memory page (4 or 8 KB) as it arrives; a response cache, a gzip filter, an application server’s own buffering, or some load balancers and CDNs hold data back. A stand-in for the worst case: a proxy that forwards only in 8 KB blocks, and that, started with --honor, respects X-Accel-Buffering: no, the response header nginx honors by default to turn buffering off for one response:

async def handle(client_r, client_w):
    up_r, up_w = await asyncio.open_connection("127.0.0.1", 8765)
    asyncio.create_task(pipe_up(client_r, up_w))
    buffered, head, passthrough = b"", None, False
    while data := await up_r.read(65536):
        if head is None:
            head = data.split(b"\r\n\r\n", 1)[0]
            passthrough = HONOR and b"x-accel-buffering: no" in head.lower()
        if passthrough:
            client_w.write(data); await client_w.drain(); continue
        buffered += data
        while len(buffered) >= BLOCK:
            client_w.write(buffered[:BLOCK]); buffered = buffered[BLOCK:]; await client_w.drain()
    client_w.write(buffered); await client_w.drain(); client_w.close()

Full script: proxy.py

The last two runs of experiments.py send the same prompt through it.

Buffering:

through the proxy: first text after 4.91 s, the end after 4.91 s, 100 pieces, 2,583 bytes

With --honor:

through the proxy: first text after 0.20 s, the end after 5.03 s, 100 pieces, 2,583 bytes

Through the buffering proxy the first text arrives after 4.91 seconds, with the last: the whole 2.6 KB answer fit in one block, so the user got none of streaming’s benefit, and the server and the client both behaved correctly. When the proxy honors the header, the first text arrives after 0.20 seconds, close to the 0.13-second average the five prompts got with no proxy at all. The header is nginx’s convention and only helps where a hop honors it; other proxies, load balancers and CDNs have their own settings, which have to be checked one by one. The other usual suspects:

• compression: gzip middleware buffers to compress well; streams must be excluded or flushed per event;
• idle timeouts: nginx closes an upstream connection that’s silent for 60 seconds by default, an AWS load balancer after 60, Cloudflare after about 100, and a long reasoning step can be that quiet. A comment line (: ping) every 15 seconds or so, as the SSE specification suggests, keeps the connection alive; Anthropic sends ping events for the same reason;
• connection limits: over HTTP/1.1 a browser opens at most six connections to one host, shared by all its tabs, and each open stream holds one. HTTP/2 and HTTP/3 multiplex streams over one connection, up to a limit the server sets, usually 100.

The test is always the same: measure the time to the first event, not the first byte (a buffering hop often passes the headers on at once), through the whole path a user’s request takes, not against the server alone.

What real streams also carry

Text deltas are the simple case. A production client also has to handle:

• Usage at the end. OpenAI sends token counts in a final chunk, when asked, whose choices list is empty, which crashes code that reads choices[0]; Anthropic’s message_delta carries cumulative usage.
• Tool calls as partial JSON. A function call’s arguments stream as fragments of a JSON string (OpenAI’s arguments deltas, Anthropic’s input_json_delta). Accumulate them and parse once the block ends, never per fragment.
• Errors after success. Once the 200 status and the headers are sent, the status can’t change, so a failure mid-stream arrives as an event: Anthropic’s error event, an OpenAI chunk with an error field.
• Partial markdown. The text arrives with unclosed code fences and half-written tables; rendering the whole answer again on every token is slow, and rendering it naively flickers.
• Flushing in the framework. FastAPI’s StreamingResponse sends each yielded piece; a hand-written server should also disable Nagle’s algorithm (TCP_NODELAY, which asyncio sets), or small events can wait tens of milliseconds for company.

Where this leads

• Serving Metrics and Prefill–Decode Disaggregation measures TTFT and TPOT under real serving load, where batching trades one against the other.
• The KV Cache explains the memory a cancelled request should free.
• System Design for AI builds a voice agent, which needs the two-way channel: the user interrupts while the model speaks.

Takeaways

• Streaming doesn’t make generation faster. It moves the wait the reader feels from the whole answer (4.7 s here for 100 tokens) to the time to first token (0.13 s).
• Readers take in about 5.5 tokens a second; generation runs several times faster, so with streaming the reader never waits after the first token, as long as time per output token stays under about 180 ms.
• Model APIs stream with server-sent events: one long HTTP response, a data: line per event, a blank line between. OpenAI’s Chat Completions end with [DONE]; Anthropic, and OpenAI’s Responses API, name every event. WebSockets add a channel back to the server.
• Send text, not tokens: decode incrementally, or characters split across tokens arrive broken.
• Stop generating when the user leaves, or a closed tab keeps the GPU and the KV cache busy: 90 of 100 tokens here. Use the transport’s disconnect signal, plus a timeout or heartbeat, since no single signal works through proxies, over HTTP/2 and for vanished clients.
• A buffering hop turns a stream back into one late response. Test the time to the first event through the whole path, and turn off buffering, compression and short idle timeouts for streams.

References

Start with the WHATWG section on server-sent events, which is short, and Anthropic’s streaming documentation for a real event schema. RFC 6455 §5 is the WebSocket framing written out.

• WHATWG, HTML Living Standard, §9.2 Server-sent events: the text/event-stream format and EventSource.
• Fette and Melnikov, RFC 6455: The WebSocket Protocol (IETF, 2011): the handshake, framing and masking.
• Anthropic, Streaming messages: the Messages API’s event types.
• OpenAI, openai-python _streaming.py and the cookbook on streaming: the [DONE] sentinel and the chunk’s fields.
• nginx, ngx_http_proxy_module: proxy_buffering: buffering, and the X-Accel-Buffering header.
• Weiss, Ayzenshteyn, Amit and Mirsky, What Was Your Prompt? A Remote Keylogging Attack on AI Assistants (USENIX Security, 2024): reading responses from the sizes of encrypted streamed tokens.
• Brysbaert, How many words do we read per minute? A review and meta-analysis of reading rate (Journal of Memory and Language, 2019): 238 words a minute for non-fiction.