I have been thinking about a logging tradeoff in LLM applications.
For debugging, the most useful thing is often the original prompt and response. But in many real systems, storing full user prompts is either not acceptable or should be avoided by default because of privacy, compliance, or just basic data minimization.
So the question becomes:
If you cannot store the prompt itself, what metadata is still useful enough for debugging?
A few fields seem relatively safe and helpful to me:
- model name
- provider
- operation name, such as
support_summaryordocument_extraction - timestamp
- latency
- status code or error category
- estimated input/output token counts
- whether streaming was used
- whether a retry happened
- whether fallback was used
- whether the output passed validation
- a request ID that can connect logs without exposing content
The hard part is that these fields can tell you what happened operationally, but not always why the model behaved badly.
For example, if a response is low quality, missing context, or malformed, prompt content would make debugging much easier. Without it, you may only know:
- the prompt was large
- the model returned valid JSON or did not
- the request hit a timeout
- the output failed a schema check
- the user retried the same task
That is useful, but incomplete.
One compromise I have seen is to log derived metadata instead of raw content, such as:
- prompt length bucket
- language
- document type
- task type
- validation failure type
- retrieval result count
- whether required variables were present
- redacted template ID instead of the final rendered prompt
But even derived metadata can leak more than expected if the categories are too specific.
Curious how others handle this.
For privacy-sensitive LLM apps, what do you keep for debugging when you cannot store user prompts?
Do you rely mostly on metadata, redacted prompt templates, short retention windows, local-only logs, or something else?