From Send to First Token: What a Streaming LLM Request Actually Does
A public-API-grounded tour of tokenization, prompt processing, autoregressive decoding, server-sent events, and the limits of what an API reveals.
On this page

A typing indicator can make an AI answer feel immediate, but it is not a transparent window into a model’s machines. The useful engineering question is narrower: what does the client send, what events can it observe, and which parts of the inference path remain provider-controlled? OpenAI’s Responses API documents streaming as server-sent events and defines events such as response.created, response.output_text.delta, response.completed, and error. [1]
A request timeline, with the private steps left private
A realistic architecture diagram should separate the client-visible contract from the service implementation:
application client
├─ constructs input and request options
├─ sends an HTTPS API request
└─ consumes a stream of server-sent events
│
▼
provider-managed service
├─ validates and schedules the request (implementation-specific)
├─ processes input context and generates output (model/runtime-specific)
└─ emits response events as they become available
The API can tell you whether streaming was requested and what response events arrived. It does not tell you which internal queue held the request, what moderation model ran, where a GPU was located, or whether a specific scheduler batched it with other requests. Those details may differ by product, model, service tier and time. Treating a familiar diagram from another inference server as a description of ChatGPT would be speculation.
What “first token” actually measures
Time to first token (TTFT) is an operational metric: elapsed time from a defined client event, often request submission, to the first usable generated content. Teams should define both endpoints precisely. If the timer begins before local serialization or after a proxy receives the request, measurements are not directly comparable. DNS, TLS setup, network distance, input size, server queueing and model processing can all contribute, but public API output usually does not expose a breakdown for every component.
Tokenization converts text into model-specific units. A token may be a complete common word, a word fragment, punctuation or another encoded piece; it is not a promise that each emitted chunk maps to one token or word. A streaming API’s delta is a transport-level piece of output. The UI should append it as text and handle event boundaries independently from word boundaries.
Input processing is often called prefill, while the subsequent autoregressive generation loop is commonly called decoding. In that loop, the model predicts a next token conditioned on the context and tokens generated so far. Implementations commonly retain key and value tensors from attention layers in a KV cache so they need not recompute the same prior-token projections for every generated token. Cache design changes memory use and scheduling trade-offs; Hugging Face documents multiple cache strategies rather than one universal implementation. [3]
These concepts describe common transformer inference mechanics. They are not evidence that a specific hosted product uses a particular cache type, batch policy, GPU topology, speculative decoder or moderation sequence.
Streaming is a protocol contract, not a speed guarantee
With server-sent events, the server sends a response body as a sequence of event records. OpenAI’s API reference specifies typed Responses events and their fields. A client can react to a text delta, wait for the completion event, and separately handle an error. [2] The SSE event-stream framing itself is specified by WHATWG HTML. [4]
request submitted
→ response.created
→ response.output_text.delta (repeated)
→ response.completed
That simplified sequence is not exhaustive: a response can contain tool calls, refusals, audio, structured output or other events. A robust client must handle unknown or irrelevant event types, interrupted connections, cancellation, timeouts and incomplete responses. Do not assume a stream always ends with a normal completion event.
Rendering early text reduces perceived waiting for long outputs, but it changes the user experience and safety boundary. OpenAI explicitly notes that streaming makes moderation harder because partial completions are more difficult to evaluate; moderation scores, when requested, arrive after the full output rather than with partial deltas. [1]
For a user-facing application, decide whether partial text can be displayed immediately, buffered for a short policy check, or replaced with a progress state for sensitive workflows. Also define what happens when the network ends mid-sentence: show that the response was interrupted, preserve or discard partial text according to product policy, and offer a retry that does not silently duplicate tool actions.
Measure the part you own
A useful client-side timing record can include request creation, connection start, first response byte, first text delta, completion, cancellation and error. Keep the API’s returned timing data distinct from client measurements. Break out network and rendering costs only when instrumentation can actually observe them.
For a multi-stage product, report prompt size, selected model, request options and streaming mode alongside latency. Avoid comparing TTFT across different input lengths or model choices as if the server-side workload were identical. A large context can affect prefill work; output length mostly affects total completion time. These are useful hypotheses to test in your own integration, not a guarantee about one provider’s hidden scheduler.
There is a parallel with ordinary API boundaries: the caller can measure the contract it observes, while the service owner controls its internal implementation. Our API-versus-SDK guide explains why client helpers do not make that ownership boundary disappear.
Do
- Define TTFT from explicit client events and report the measurement method with the number.
- Treat streamed deltas as partial text; handle completion, errors, cancellation and disconnects.
- Cite public API behavior separately from general transformer mechanics and provider-private infrastructure.
Don't
- Do not claim that a public endpoint proves the provider’s internal moderation, routing or GPU pipeline.
- Do not equate one stream event with one word or one model token.
- Do not assume a successful first delta means the complete response or a tool action finished.
Streaming changes when users can begin reading; it does not turn hidden serving details into public facts. Build around the events and measurements the API actually exposes, label the rest as implementation-specific, and make interrupted output an explicit state.
Frequently asked questions
Does a streaming API send one word at a time?
Not necessarily. The API emits text deltas as generation proceeds, but each delta can contain a varying amount of text. A token is a model vocabulary unit and does not always correspond to one word.
What is the difference between prompt prefill and decoding?
Prefill processes the input context for generation. Autoregressive decoding then produces output sequentially, with each next-token step conditioned on prior context and generated tokens. Implementations can batch or schedule work differently.
Does the OpenAI API document ChatGPT's internal serving pipeline?
No. Public API documentation describes request parameters and observable response events. It does not establish ChatGPT's private moderation, routing, GPU placement, batching, or internal service topology.
Can a client moderate streamed output before showing it?
A client can inspect chunks, but moderation over partial output has different semantics from evaluating a completed response. OpenAI warns that streaming makes moderation more difficult; applications should design a policy for partial content and interrupted streams.



/* Comments */
Comments are offline right now — we reconnect automatically, nothing is lost.