Streaming LLM responses with Turbo Streams

An LLM call is slow. Not slow like a database query that takes 50ms — slow like several seconds, sometimes ten or more for a long completion. If you make the user wait for the whole response before showing anything, they sit watching a spinner for the full duration, and the feature feels broken even when it’s working. The fix is streaming: the model emits its output token by token, and you push each chunk to the page as it arrives, so the user watches the answer appear like someone typing rather than waiting for a wall of text to drop in at the end. The model is no faster — but the experience is transformed, because dead wait becomes live progress.

And here’s the satisfying part for a Rails developer: pushing incremental HTML to a live page is exactly what Turbo Streams was built to do. We’ve spent years in these posts on the “HTML over the wire” idea — the server renders HTML and ships it to the browser, which splices it into the DOM. An LLM streaming its output is just that pattern with a new source of chunks. Let me walk through how the pieces fit.

Why streaming, concretely

The case for streaming is about perceived latency, which is the only latency the user feels:

• A spinner for 8 seconds feels broken. Users assume something hung. Even a progress bar barely helps because nothing is happening that they can see.
• Text appearing within a second, then flowing, feels alive. The total time to the complete answer is the same — but the user got feedback immediately and watched it build, so it feels fast and responsive even though the model took just as long.
• It sets the right expectation. Watching tokens stream tells the user “this is being generated for you right now,” which is honest about what’s happening and what kind of latency to expect.

This is the same lesson as every other perceived-performance trick: the user’s sense of speed is about feedback, not raw throughput. Streaming gives feedback continuously instead of once at the end.

The model end: tokens as they generate

The OpenAI API (and others) supports a streaming mode: instead of returning the whole completion in one response, it streams the output as a series of server-sent events, each carrying the next small chunk of text. In Ruby, you consume it as a stream and get a callback per chunk:

client.chat(
  parameters: {
    model: "gpt-3.5-turbo",
    messages: [{ role: "user", content: question }],
    stream: proc do |chunk, _bytesize|
      token = chunk.dig("choices", 0, "delta", "content")
      broadcast_token(token) if token
    end
  }
)

Each token is a small piece of the answer — sometimes a word, sometimes part of one. Our job is to get each one onto the user’s screen as it arrives. That’s where Rails comes in.

The Rails end: Turbo Streams pushes each chunk

Turbo Streams lets the server send a fragment of HTML and an action (“append this to that element”) over a persistent connection, and Turbo on the page applies it. For LLM streaming, we want to append each token to a target element as it comes in:

def broadcast_token(token)
  Turbo::StreamsChannel.broadcast_append_to(
    "answer_#{@conversation.id}",
    target: "answer_body",
    html: token
  )
end

On the page, we subscribe to that stream and give the target a home:

<%= turbo_stream_from "answer_#{conversation.id}" %>
<div id="answer_body"></div>

Now every token the model emits gets appended to #answer_body the instant it’s generated, and the user watches the answer type itself out. No page reload, no polling, no custom JavaScript — the same Turbo Streams machinery that broadcasts a new comment to a live page is broadcasting model tokens. The “new source of chunks” really is the only new thing.

Run it in a background job

The streaming call takes seconds and holds open a connection to the model the whole time. That does not belong in a web request — tying up a Puma worker (or worse, a request thread) for ten seconds per question is exactly the kind of slow, blocking I/O that background jobs exist to absorb. So the shape is:

1. The user submits a question. The controller enqueues a job and immediately returns, rendering an empty answer container and subscribing it to the Turbo Stream.
2. The job runs the streaming model call, broadcasting each token to that stream as it arrives.
3. The user, already looking at the (initially empty) container, watches it fill in real time.

class AnswerJob < ApplicationJob
  def perform(conversation)
    StreamingAnswer.new(conversation).call  # broadcasts tokens as it streams
  end
end

This is the same separation we’ve reached for again and again: the web request stays fast and does the minimum, and the slow external call happens in a worker. Streaming doesn’t change that principle — if anything it makes it sharper, because the call is now both slow and long-lived.

Closing the stream cleanly

When the model finishes, the stream of tokens ends, and you usually want to do a little housekeeping: mark the answer complete, persist the full text, maybe swap a “generating…” indicator for a done state. Append a final broadcast that replaces the container or toggles a flag:

def finish
  @conversation.update!(answer: @buffer, status: :complete)
  Turbo::StreamsChannel.broadcast_replace_to(
    "answer_#{@conversation.id}",
    target: "answer_body",
    partial: "answers/complete",
    locals: { conversation: @conversation }
  )
end

Note that you also accumulate the tokens into a buffer as you stream, so that when it’s done you have the complete text to persist. The user saw it appear live; the database gets the final version. Both matter — the live view for the experience, the saved copy so the answer survives a page reload.

What can go wrong, and handling it

Streaming adds a couple of failure modes worth designing for:

• The model call fails partway. You’ve streamed half an answer and the connection drops or the API errors. Broadcast an error notice into the same container and mark the conversation failed — don’t leave a half-answer sitting there looking complete. Because you’re in a background job, your normal retry logic applies, but be careful: retrying a partially streamed answer means the user might see it start over, so decide deliberately whether to retry or to surface the failure.
• The user navigates away. The job keeps running and broadcasting to a stream nobody’s listening to, which is harmless but wasteful for a long generation. Not usually worth solving early, but worth knowing.
• Tokens arrive faster than they render. Rarely a problem at LLM speeds (the model is the bottleneck, not the browser), but if you ever batch tokens for efficiency, append in small groups rather than one element per token to keep the DOM churn down.

None of these are exotic — they’re the ordinary failure-handling of any slow external call, which is exactly how we’ve learned to treat the LLM API throughout this series: a slow, metered, sometimes-failing external service that you call from a job and whose failures you handle explicitly.

Verdict

LLM responses are slow enough that making the user wait for the whole thing feels broken — and streaming the output token by token fixes the experience without making the model any faster, by turning a dead spinner into live, flowing text. The beautiful part for Rails developers is that you already have the perfect primitive: Turbo Streams was built to push incremental HTML to a live page, and an LLM emitting tokens is just a new source of chunks for that same machinery. Consume the model’s stream in a background job (because it’s slow and long-lived, exactly where slow I/O belongs), append each token to a target element via broadcast_append_to, accumulate a buffer so you can persist the complete answer, and broadcast a clean finish — handling the partial-failure case explicitly like any other external call. The model is the same speed it always was; the user just gets to watch it think. That’s the whole trick, and it’s a small amount of code on top of patterns you already know.