How do you log LLM API calls without storing user prompts?

AI Discussions
1

I’ve been working on a small API infrastructure project recently, and one question keeps coming up:

How much should we log for LLM API calls without accidentally storing sensitive user data?

For normal API calls, I usually log things like status code, latency, request ID, and error type.

For LLM calls, that has not been enough. A request can technically “succeed” while still being problematic:

  • latency gets much worse
  • retries happen silently
  • token usage increases over time
  • fallback models start handling more traffic than expected
  • streaming starts but does not complete
  • long prompts fail more often than short ones

The obvious debugging answer is “log the prompt,” but that feels risky in production. Prompts may contain private user messages, customer documents, internal business logic, or data that should not be kept longer than necessary.

The compromise I am currently considering is logging metadata like:

  • model
  • provider
  • request ID
  • feature or product area
  • latency
  • input/output token count
  • retry count
  • fallback source/target
  • error category
  • prompt hash instead of raw prompt
  • message count
  • approximate input length

For example:

{
  "event": "llm_request",
  "request_id": "req_123",
  "feature": "support_reply_generator",
  "model": "gpt-4.1-mini",
  "status": "success",
  "latency_ms": 1842,
  "input_tokens": 812,
  "output_tokens": 244,
  "retry_count": 0,
  "fallback_from": null,
  "fallback_to": null,
  "prompt_hash": "a3f9c01de81b7a22"
}

The part I am still unsure about is where the line should be.

Hashing prompts helps identify repeated failures, but it does not help much when debugging why the model behaved badly. Redaction helps, but it can be brittle. Short retention windows help, but they do not solve access-control issues.

For people building or operating LLM applications in production:

  • Do you log raw prompts at all?
  • Do you hash, redact, or sample prompts?
  • What fields have actually helped you debug model/API issues?
  • How do you balance observability with privacy and retention concerns?

I would be curious to hear what practices people have found sustainable.

2

I’m not an expert in production LLM systems, but from a statistical/ML perspective, a common approach is to anonymize or de-identify collected data before analysis. This helps reduce privacy risks while still preserving useful signals for evaluation and debugging. With proper security controls, this balance is often used to manage privacy vs. observability.