How do you log LLM API calls without storing user prompts?

I’ve been working on a small API infrastructure project recently, and one question keeps coming up:

How much should we log for LLM API calls without accidentally storing sensitive user data?

For normal API calls, I usually log things like status code, latency, request ID, and error type.

For LLM calls, that has not been enough. A request can technically “succeed” while still being problematic:

  • latency gets much worse
  • retries happen silently
  • token usage increases over time
  • fallback models start handling more traffic than expected
  • streaming starts but does not complete
  • long prompts fail more often than short ones

The obvious debugging answer is “log the prompt,” but that feels risky in production. Prompts may contain private user messages, customer documents, internal business logic, or data that should not be kept longer than necessary.

The compromise I am currently considering is logging metadata like:

  • model
  • provider
  • request ID
  • feature or product area
  • latency
  • input/output token count
  • retry count
  • fallback source/target
  • error category
  • prompt hash instead of raw prompt
  • message count
  • approximate input length

For example:

{
  "event": "llm_request",
  "request_id": "req_123",
  "feature": "support_reply_generator",
  "model": "gpt-4.1-mini",
  "status": "success",
  "latency_ms": 1842,
  "input_tokens": 812,
  "output_tokens": 244,
  "retry_count": 0,
  "fallback_from": null,
  "fallback_to": null,
  "prompt_hash": "a3f9c01de81b7a22"
}

The part I am still unsure about is where the line should be.

Hashing prompts helps identify repeated failures, but it does not help much when debugging why the model behaved badly. Redaction helps, but it can be brittle. Short retention windows help, but they do not solve access-control issues.

For people building or operating LLM applications in production:

  • Do you log raw prompts at all?
  • Do you hash, redact, or sample prompts?
  • What fields have actually helped you debug model/API issues?
  • How do you balance observability with privacy and retention concerns?

I would be curious to hear what practices people have found sustainable.

I’m not an expert in production LLM systems, but from a statistical/ML perspective, a common approach is to anonymize or de-identify collected data before analysis. This helps reduce privacy risks while still preserving useful signals for evaluation and debugging. With proper security controls, this balance is often used to manage privacy vs. observability.

I would make *content-free telemetry* the default and treat prompt capture as a separate, exceptional debugging workflow. Your proposed fields are a good start; I’d adjust the schema so it explains the silent failures you listed without needing the message text.

For each logical request, record an internal trace/request ID, feature, provider/model (including the model actually used after fallback), deployment/config version, retry/attempt number, timeout/error class, final outcome, input/output token usage when reported, and whether the stream completed normally. Keep timestamps for request start, time-to-first-token and stream end; total latency alone will not distinguish slow first tokens from a stalled stream. A low-cardinality prompt-size bucket (and message count) can support length-related analysis without recording the prompt.

Two changes I would make to your proposal:

  • **Do not default to `prompt_hash`.** An unkeyed hash of predictable text can be tested against a dictionary of likely prompts; it also creates a stable cross-request identifier. If duplicate detection is truly necessary, assess a keyed HMAC with protected key material, scoped retention and restricted access instead. Otherwise use a random request/trace ID and store no prompt fingerprint.
  • **Keep debug logs separate from metrics.** Put model, outcome and latency/token histograms into aggregated telemetry; keep individual request IDs and error details in access-controlled traces. Scrub exceptions and HTTP headers because prompts or tokens can leak there even when you never intentionally log `messages`.

For a successful-but-wrong answer, metadata can tell you *which* model/config handled it and how, but it cannot reliably explain the model’s semantics. I would reproduce issues with synthetic or explicitly approved test cases, plus a versioned evaluation set. If production-content capture is ever warranted, require explicit authorization, narrow incident scope, encryption, audited access, defined deletion, and review of vendor/collector retention.

This aligns with OpenTelemetry’s GenAI observability approach, which supports model, latency and token metrics and treats content capture as something enabled deliberately: Inside the LLM Call: GenAI Observability with OpenTelemetry | OpenTelemetry . OWASP also recommends excluding or tightly protecting sensitive personal data, tokens and secrets from application logs: Logging - OWASP Cheat Sheet Series .