I have been thinking about a logging/debugging tradeoff in LLM apps.
For simple prototypes, it is tempting to log the full prompt and response whenever something fails. It makes debugging much easier.
But in real products, that can be a bad default. User prompts may contain private data, company context, customer text, or just things you do not want sitting in logs forever.
So the question becomes:
If you decide not to store raw user prompts, what metadata is still useful enough for debugging?
The fields I usually think about are things like:
- model name
- provider name
- request ID
- latency
- status code
- error type
- retry count
- token usage
- finish reason
- whether streaming completed
- whether a fallback was used
- a short internal error category
- maybe a hashed prompt fingerprint for grouping similar failures
The tricky part is that some failures are hard to understand without seeing the actual input.
For example:
- context length failures
- bad tool-call arguments
- refusal or policy-related behavior
- malformed JSON output
- weird routing/fallback behavior
- user reports saying “the answer was wrong”
In those cases, metadata helps, but only up to a point.
One compromise I have seen is logging structured metadata by default, then allowing short-term, explicit, access-controlled prompt capture only for debugging specific incidents. Another option is to log redacted prompt summaries, but that can also hide the exact thing that caused the failure.
I am curious how others handle this.
If you are building or evaluating LLM apps, do you log raw prompts at all? Or do you rely mostly on metadata, traces, user reports, and reproducible test cases?