I’m trying to think more clearly about latency when using streaming LLM responses, and I’m curious how others here measure it.
For normal API calls, latency is fairly straightforward: request starts, response completes, measure total time.
With streaming LLM responses, I’m finding that one number is not enough.
For example, a response can feel fast if the first token appears quickly, even if the full answer takes a long time. But for some workflows, the total completion time matters more than first-token latency.
Right now I’m separating it into a few measurements:
- time to first token
- time to first useful sentence
- total generation time
- tokens received before interruption
- whether the stream completed cleanly
- whether the user abandoned the interaction before completion
The part I’m still unsure about is how to compare models or providers fairly.
A model with slower first-token latency might produce a more complete answer faster overall. Another model might stream quickly but take longer to reach a usable final response. And for structured output, streaming may not help much if the app cannot use the result until the full JSON object is valid.
So I’m wondering how others approach this in practice.
When you evaluate streaming LLM latency, do you mainly care about time to first token, total completion time, perceived user experience, or something else?
And do you measure streaming differently for chat interfaces versus background workflows?