What do you log when an LLM stream fails halfway through?

AI Discussions
,
1

I’ve been thinking about logging for streamed LLM responses, especially the cases where the response starts normally and then fails halfway through.

For regular API calls, I usually log things like status code, latency, request id, and error type. But streamed LLM calls feel different because the failure can happen at several points:

  • the request is rejected before the stream starts
  • the provider accepts the request but never sends the first token
  • the first tokens arrive, then the connection drops
  • a tool call starts streaming but the arguments are incomplete
  • the model seems to finish, but no usage or final event is received
  • the client closes the connection before the upstream stream finishes

Those cases all look different operationally, but they often get collapsed into something vague like stream failed.

The fields I’m considering logging are:

  • whether the first token was received
  • time to first token
  • number of chunks received
  • approximate output length received before failure
  • whether a final done event was received
  • whether finish_reason was available
  • whether usage metadata was available
  • whether the failure happened before or after tool call generation
  • whether the client or upstream provider closed the connection

I’m also trying to avoid logging raw prompts or raw model outputs unless absolutely necessary. So the hard part is getting enough debugging signal without storing sensitive user content.

For people running streamed LLM responses in production:

Do you classify failures by stream stage, such as “before first token” vs “after first token”?

And if you avoid storing raw prompts/responses, what metadata has actually been useful for debugging mid-stream failures?