Streaming server-sent events through nginx and Cloudflare without buffering

Catalyst streams LLM responses as server-sent events from FastAPI through nginx and Cloudflare, and in October 2024 they arrived late and in chunks because nginx was buffering them; proxy_buffering off fixed that. Since then the app also sends X-Accel-Buffering: no and Cache-Control: no-cache, no-transform with every stream, so streaming doesn’t depend on the proxy config. It also sends a keepalive after 15 seconds of silence, so a long tool call doesn’t get its connection cut.

Updated October 2026: this post now explains what each nginx line actually does, and it adds the response headers for Cloudflare and the keepalives Catalyst has needed since.

How I found it (October 2024)

The API is FastAPI, run by Gunicorn with Uvicorn workers on a Unix socket, behind nginx on an EC2 instance, with Cloudflare proxying the domain. (The first version of this post called Gunicorn the WSGI server. FastAPI is an ASGI app, and the Uvicorn workers are what run it.) The responses came from OpenAI’s GPT models then.

Through the proxied domain, responses came through slowly, in delayed chunks. I turned off Cloudflare’s proxy for a while and it was still slow, so Cloudflare wasn’t the cause. When I called the app directly on port 8000 on the instance, it streamed smoothly. That left nginx.

The nginx location

This is the block from 2024, and its streaming lines are still what runs. The one change came in July 2026, when the Connection header started coming from a map so WebSockets can upgrade. For ordinary requests it’s still empty.

location /api/ {
    proxy_set_header Host $http_host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;

    proxy_pass http://unix:/path/to/app.sock;

    proxy_buffering off;
    proxy_cache off;
    proxy_set_header Connection '';
    proxy_http_version 1.1;
    chunked_transfer_encoding off;
    proxy_read_timeout 300s;
    proxy_send_timeout 300s;
    gzip off;
}

In 2024 I listed six changes as if they all mattered equally. They don’t.

• proxy_buffering off fixes the delay. By default nginx reads the app’s response into buffers. With buffering off, nginx’s docs say “the response is passed to a client synchronously, immediately as it is received.”
• proxy_cache off is the default anyway. It’s here so a cache set at a higher level can’t apply to this location.
• proxy_http_version 1.1 and the empty Connection header. I called these “keepalive connections”. They are what nginx needs before it can reuse upstream connections. But on nginx 1.24, the version on that server, it only reuses them when an upstream block has a keepalive directive, and this config has none. What 1.1 does here is let the app send a chunked response. Over HTTP/1.0, which 1.24 uses unless told otherwise, the app would have to close the connection to end a stream. (nginx 1.29.7 changed both defaults: it now proxies with HTTP/1.1 and keeps upstream connections alive.)
• chunked_transfer_encoding off changes how nginx frames the response to the client, not how fast it sends it. The HTML standard notes that chunking “can have unexpected negative effects on the reliability” of event streams and that it can be disabled. With it off, nginx has no length to go by, so it closes the connection when the stream ends.
• proxy_read_timeout 300s is the longest nginx waits between two reads from the app, not a limit on the whole response. The default is 60 seconds.
• gzip off. My nginx.conf has gzip on with the default gzip_types, which covers only text/html, so event streams weren’t being compressed anyway. This line keeps it that way if the list ever grows.

Headers from the app (February 2026)

When Catalyst moved generation into server-side jobs with a stream per job, in February 2026, the stream endpoint started sending X-Accel-Buffering: no. A week later it added no-transform and Content-Encoding: none for Cloudflare:

return StreamingResponse(
    event_generator(),
    media_type="text/event-stream",
    headers={
        "Cache-Control": "no-cache, no-transform",
        "X-Accel-Buffering": "no",
        "Content-Encoding": "none",
    },
)

• X-Accel-Buffering: no turns off nginx buffering for this one response, even in a location where proxy_buffering is on. nginx’s docs say buffering “can also be enabled or disabled by passing “yes” or “no” in the “X-Accel-Buffering” response header field.” The stream no longer depends on someone getting the location block right.
• Cache-Control: no-transform is for Cloudflare. Its cache docs say “Compression is disabled when the no-transform directive is present”. The same goes for other features that rewrite a body, such as Polish and JavaScript Detections. text/event-stream isn’t on Cloudflare’s list of types it compresses by default, so this mostly protects against a later setting changing that. no-cache tells caches, Cloudflare included, not to serve a stored copy without revalidating it.
• Content-Encoding: none is the one I can’t back with documentation. none isn’t in IANA’s registry of content codings, and Cloudflare’s docs don’t say what it does with an unknown one. I added it to keep Cloudflare from compressing or buffering the stream. no-transform is the documented way to ask for the first.

If a stream still arrives late or all at once behind Cloudflare, check Configuration Rules. They have had a response body buffering setting since January 2026. The default, Standard, lets Cloudflare products inspect the start of a response body. None streams the body straight to the client, at the cost of features that inspect bodies, like the WAF.

Keepalives for quiet streams (July 2026)

An LLM response isn’t always talking. When a model calls a tool that takes minutes, such as image generation, the stream goes silent. nginx closes the upstream connection after proxy_read_timeout passes with no data, Cloudflare documents a proxy read timeout of 125 seconds, and our own web client gives up after 60 seconds without a frame. In July 2026 those silences showed up as “Connection lost” and a Retry button, while the job itself was still running fine.

The code was supposed to send keepalives already. It sent them when asyncio.wait_for raised a timeout, but redis-py’s get_message(timeout=1.0) returns None after a second of silence instead of raising. So the keepalive branch never ran. Now the clock drives it:

SSE_KEEPALIVE_SECONDS = 15.0

last_yield = time.monotonic()
while not await request.is_disconnected():
    msg = await pubsub.get_message(ignore_subscribe_messages=True, timeout=1.0)
    if msg and msg["type"] == "message":
        yield f"data: {msg['data']}\n\n"
        last_yield = time.monotonic()
    elif time.monotonic() - last_yield >= SSE_KEEPALIVE_SECONDS:
        yield f"data: {json.dumps({'keepalive': True})}\n\n"
        last_yield = time.monotonic()

The response stream sends a small JSON event that the clients ignore, apart from resetting their idle timers. Catalyst’s per-user event channel uses an SSE comment instead, a line that starts with a colon:

: keepalive

EventSource ignores comment lines by spec. The HTML standard suggests exactly this for proxies that drop idle connections: “a comment line (one starting with a ‘:’ character) every 15 seconds or so.” Fifteen seconds stays well under all three limits.

How to check

Timestamp each line as it arrives, once through Cloudflare and nginx and once straight to the app on the server:

curl -sN -H "Authorization: Bearer $TOKEN" \
  "https://api.example.com/api/streaming/jobs/$JOB/stream" |
  while IFS= read -r line; do printf '%s %s\n' "$(date +%T)" "$line"; done

curl -sN --unix-socket /path/to/app.sock \
  -H "Authorization: Bearer $TOKEN" \
  "http://localhost/api/streaming/jobs/$JOB/stream"

Events should spread over the seconds the model takes, the same on both paths. If the proxied one prints nothing and then everything at once, something on the path is buffering. To see the headers, add -D - -o /dev/null --max-time 5. You want content-type: text/event-stream and no content-encoding: gzip or br.

What I’d tell myself in 2024

Isolate layer by layer, as I did: through the CDN, around the CDN, straight to the app. Then put the fix in the response as well as in the proxy. Headers travel with the stream, but a config file belongs to whoever edits it next. And plan for silence as well as speed. A stream that goes quiet for a few minutes looks dead to something between the model and the reader.