Streaming AI text without the jank

A model produces text a token at a time. A token is a piece of a word, often a few characters. The server sends them down an open HTTP response, usually as server-sent events or a streamed fetch body that you read chunk by chunk.

On the way, tokens get grouped. The server may batch them, proxies and TCP buffer them, and the browser hands you whatever has arrived when it next gets round to it. So you receive one chunk with a single character, and the next with forty.

The naive renderer does this:

for await (const chunk of stream) {
  text += chunk;
  output.innerHTML = markdown(text);
}

It has three separate problems. The rhythm is the network's, not a reader's. The parse gets slower as the answer grows. And for a moment, half-finished Markdown renders as the wrong thing.

The fix for rhythm is to stop letting the network decide when text appears.

1. Network uneven chunks
2. Buffer holds what's arrived
3. Reveal clock every frame
4. Incremental parser last block only
5. DOM append, don't replace

Chunks go into a buffer the moment they land. Separately, a loop running on requestAnimationFrame reveals a few characters each frame. Ten chunks in one frame become one update, so you never render more often than the screen can show.

The interesting part is the speed. A fixed rate doesn't work. Too slow and the display falls further and further behind a fast model. Too fast and it's the staircase again.

So the rate follows the backlog:

const backlog = received.length - shown;
const lag = streamEnded ? 0.12 : 0.35;      // seconds we'll sit on
const rate = Math.max(30, backlog / lag);    // characters per second
budget += rate * dt / 1000;
const step = Math.min(backlog, Math.floor(budget));

When a big chunk lands, the backlog jumps and the clock speeds up to catch up. As it drains, it slows down. The lag is how much text you're willing to hold back to smooth things out, and it's a feel decision. More lag rides through longer stalls but makes the answer feel slower to start. When the stream ends, the lag drops so the last words don't trickle in.

The clock can't invent text. If the network stalls for a second, the red line flattens too. Smoothing evens out the bursts. It can't hide a slow model.

Re-parsing the whole answer on every chunk costs more each time. The 50th chunk re-reads everything the first 49 did. Over a whole answer, the work grows with the square of its length. Chrome's guide to rendering streamed LLM responses warns about exactly this pattern.

The readout shows where that goes. By the end, every naive update re-parses the whole answer. Every chunk pays for all the chunks before it.

The way out is to notice that Markdown is made of blocks, and a block is finished once a blank line follows it, unless you're inside a code fence. A finished block can never change. So you parse it once, append its HTML, and never touch it again.

Only the last, unfinished block gets re-parsed, and only on frames where new text is revealed. However long the answer gets, the work per update is one block. In the demo, that's the final sentence instead of the whole answer.

Here's the flicker. The model has sent **sea. A correct parser sees two literal asterisks, because there's no closing pair yet. Four characters later, **search boxes** arrives, and the asterisks vanish and the text turns bold. Something already on screen just changed type.

The demo counts these moments as flickers. It compares what kind of element every visible character ended up in, frame to frame.

There are two fixes, and they handle different cases.

Close it optimistically. If the tail has an open ** or backtick, render it as if it were already closed. **sea shows as bold "sea", which is what it's about to be anyway. An unclosed code fence already works this way, since everything after it is code until it closes.

Hold back what's ambiguous. Some characters genuinely can't be decided yet. A line that's just 1 might become 1. A list item or 1.5 seconds. A lone ` might open a code block. A single trailing * might be the start of bold. These wait one more character before appearing.

const last = text.slice(text.lastIndexOf("\n") + 1);
if (/^(\d+\.?|-|`{1,2})$/.test(last)) hide(last);      // block marker?
else if (/[^*]\*$/.test(last)) hide(1);                 // start of **?

Holding back one character is invisible at reading speed. Showing the wrong element and then swapping it isn't.

One more rule from the Chrome guide: if you render HTML, sanitise the whole accumulated output, not each chunk, because something dangerous can be split across two chunks.

The other way streaming goes wrong is the scroll. Following new text is right while you're at the bottom. It's wrong the moment you scroll up to reread something. It's one of the states I'd check first in any agent interface.

The logic is a small state machine with one flag, stuck:

• New text arrives. If stuck, set scrollTop to the bottom.
• The reader scrolls up, with a wheel, touch, keys or the scrollbar. Set stuck to false straight away.
• The reader scrolls back to within about 32px of the bottom. Set stuck to true.

The detail that's easy to miss is how you tell the reader's scroll from your own. Your code only ever scrolls down, to the bottom. So any upward scroll came from the reader. Listening for wheel and touchmove as well means even a small nudge up counts immediately, before the next chunk can drag them back.

Two more things change the feel. Reserve space where you can, so the rest of the page doesn't shift as the answer grows. The first demo sets each pane's height to the final answer's height up front. And under prefers-reduced-motion, make the jump to latest instant rather than smooth.

Don't put aria-live on the message that's streaming. Every change to a live region is queued for announcement, so a polite region will try to read out a stream of tiny fragments. That's also why the demo's scroll box isn't role="log", which is polite by default.

MDN suggests aria-busy to hold back announcements while a region is still updating, but screen reader support for it varies. I'd use a separate, quiet status region. It says "Responding" when the answer starts and "Response ready" when it ends, and the reader can move into the text itself whenever they like.

That's the whole pipeline. The stream is an implementation detail, and nobody reading should be able to tell how it was chunked. What they should see is steady text from something that never changes its mind about what it already showed them.