How do you measure latency for LLM APIs beyond total response time?

AI Discussions
,
1

I’ve been thinking about how teams measure latency for LLM API calls in production.

A lot of dashboards seem to start with one number: total response time.

That is useful, but I’m finding it too blunt. Two requests can both take 20 seconds and feel completely different:

  • one starts streaming tokens after 800ms and finishes slowly
  • one stays silent for 18 seconds and then returns everything at once
  • one is fast on the first attempt but slow after retries
  • one looks slow because the model is slow
  • one looks slow because the input was huge
  • one looks slow because the request sat behind rate limits or provider-side queueing

So I’m trying to separate “LLM latency” into a few more specific measurements.

The rough breakdown I’m using right now:

  1. Connection time
    How long it takes to establish the request. If this is high, I’d look at network path, DNS, TLS, client configuration, or regional routing before blaming the model.

  2. Time to first token
    For streaming responses, this is usually the metric that matters most to the user. If the first token arrives quickly, the product feels responsive even if the full response takes longer.

  3. Total completion time
    Still important, especially for background jobs, document analysis, batch extraction, and workflows where the final result is what matters.

  4. Tokens per second
    Useful, but only if compared within similar task types. A short answer, a long summarization task, and a reasoning-heavy request are not really comparable.

  5. Idle gaps during streaming
    This one is easy to miss. A stream can start quickly but then pause for several seconds in the middle. From a user’s perspective, that can feel broken.

  6. Retry-adjusted latency
    If a request fails once and succeeds on retry, the user experienced the full end-to-end delay, not just the successful attempt. I think this should be tracked separately from single-attempt latency.

  7. Fallback rate and fallback latency
    If the primary model or provider fails and the system falls back to another one, that may save the request, but it also changes latency and possibly output behavior. I’m starting to think fallback rate should be treated as an operational metric, not just an implementation detail.

One thing I’m still unsure about is how much of this should be exposed in product analytics versus internal observability.

For example, a product team may only care that “the user waited 14 seconds.” But an engineering team needs to know whether that was caused by first-token delay, slow generation, retry behavior, provider degradation, or our own queue.

I’m also not sure what a good baseline looks like across different task types. Chat, code generation, document summarization, agent workflows, and structured extraction probably need different latency expectations.

Curious how others here handle this:

  • Do you track time to first token separately from total response time?
  • Do you include retries in your user-facing latency metric?
  • Do you monitor idle gaps in streaming responses?
  • Do you treat fallback usage as part of reliability metrics?
  • Are there any latency measurements you found useful in production that are easy to overlook?