Building a Privacy-Preserving LLM Usage Analytics System (Clio)

anthropic · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Privacy-preserving LLM usage analytics and safety evaluation

Anthropic needed to understand real-world Claude usage well enough to improve evaluations, safety monitoring, refusal calibration, and product decisions without having humans read raw user conversations.

Constraints:

• top-down evaluations and red teaming can miss how users actually interact with general-purpose models
• raw conversations are sensitive and should not be inspected directly by humans for broad analytics
• the system must preserve privacy while still yielding useful aggregate insight
• cluster outputs must avoid private or identifying details
• post-deployment monitoring must complement pre-deployment testing for new capabilities such as computer use

Architecture

Clio samples real Claude conversations, uses Claude to summarize each conversation into a private high-level request, embeds those summaries, clusters semantically related conversations, uses Claude again to name and describe clusters, audits clusters for privacy risk, applies minimum aggregation thresholds for organizations and conversations, and exposes only high-level aggregate clusters internally for evaluation and safety work.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  conversation-sampler[Conversation sampler] --> claude-summarizer[Claude summarizer]
  claude-summarizer --> embedding-pipeline[Embedding pipeline]
  embedding-pipeline --> clusterer[Clusterer]
  clusterer --> cluster-describer[Cluster describer]
  cluster-describer --> privacy-auditor[Privacy auditor]
  privacy-auditor --> aggregation-gate[Aggregation gate]
  aggregation-gate --> internal-analysis-layer[Internal analysis layer]
  internal-analysis-layer --> evaluation-design[Evaluation design]
  internal-analysis-layer --> trust-and-safety-workflows[Trust and safety workflows]
  privacy-auditor --> clio-privacy-evaluation[Clio privacy evaluation]
  clio-privacy-evaluation --> evaluation-design

Components

Data flow

A random sample of Claude conversations is summarized by Claude rather than read by humans. The raw conversations are discarded for the analytics pipeline, summary embeddings are clustered by semantic content, Claude names and describes each cluster while avoiding private details, a privacy auditor checks whether cluster information could identify fewer than roughly a thousand people, quantitative aggregation minimums are applied, and only aggregate cluster summaries and statistics are exposed internally.

Why this architecture

Anthropic chose an aggregate, model-mediated pipeline because raw conversation inspection would create a privacy risk, while purely top-down evals missed real-world use. Summaries and clusters preserve enough semantic information for safety and evaluation work; privacy auditing and aggregation thresholds reduce identifiability; and synthetic examples let the team improve models without training on raw user conversations.

Models used: Claude · embedding model · privacy auditor model

Scale

• The privacy bar is calibrated around information that could identify fewer than roughly a thousand individuals.
  “is anything that could be identifying to the order of, you know, roughly a thousand individuals. And finally we apply sort of quantitative aggregation minimums.”transcript, stated
• A final aggregate cluster might represent around a thousand conversations if the privacy and size thresholds are met.
  “that includes, you know, say a thousand conversations about web development, maybe in Elixir if there are enough, and summary statistics about, say for example, the language breakdown of that cluster.”transcript, stated
• Clio validation used a synthetic corpus of tens of thousands of conversations.
  “is we generate a huge synthetic corpus of tens of thousands of conversations, and we do it through a process where we know what the ground truth distribution looks like.”transcript, stated
• The synthetic validation included known distribution targets such as 10% math, 5% coding, and 2% teddy-bear questions.
  “We know that this should be like, 10% math content, 5% coding, 2% questions about teddy bears. And we just give all of that to Clio without telling it how these conversations should be grouped.”transcript, stated

Alternatives considered (roads not taken)

Design safety evaluations only from top-down hypotheses

Rejected because: The team wanted evaluations grounded in real-world use cases rather than only hypothetical harms.

Have humans inspect raw conversations for broad analytics

Rejected because: Clio was designed so humans do not need to read raw conversation data.

Expose individual-level or small-cluster data internally

Rejected because: Clusters pass through privacy auditing and minimum aggregation thresholds before exposure.

Train directly from user conversations to reduce over-refusals

Rejected because: Anthropic instead uses topics from Clio and hires people to generate data or creates synthetic data in those domains.

Patterns

privacy-preserving-processing (analytics)

Clio processes conversations into private summaries, discards raw conversations from the analytics path, audits clusters, and exposes only sufficiently aggregated outputs.

batch-offline-processing (analytics-pipeline)

The system samples conversations, summarizes them, embeds them, clusters them, and then generates aggregate reports for downstream use.

llm-as-judge (privacy-audit)

Claude is used to inspect clusters for private or identifying details before outputs become visible internally.

eval-driven-development (evaluation)

Clio grounds new evaluations in empirical usage and validates its own privacy auditor and clustering behavior through targeted evaluations.

synthetic-data-generation (privacy-preserving-training)

Anthropic uses synthetic data both to evaluate Clio and to improve refusal behavior in domains identified by aggregate usage patterns without training on actual conversations.

Tradeoffs

Analyze aggregate clusters instead of raw conversations

Chose High-level summaries, embeddings, clusters, and aggregate statistics over Human review of raw user conversations.

Rationale: The aggregate pipeline preserves useful signal about use cases while keeping humans far from private conversation content.

Require privacy and aggregation gates before internal exposure

Chose Model privacy audit plus distinct organization and conversation minimums over Exposing clusters immediately after semantic clustering.

Rationale: Even summarized clusters can identify small groups unless both content and size thresholds are enforced.

Use Clio topics to generate new training data

Chose Human-generated and synthetic data in domains identified by clusters over Using the actual user conversations for model improvement.

Rationale: This lets Anthropic improve refusal behavior while respecting user privacy.

Combine top-down and bottom-up safety approaches

Chose Pre-deployment tests plus Clio post-deployment monitoring over Relying on either policy/eval design or production observation alone.

Rationale: Safety teams need both anticipated risks and empirical blind-spot discovery.

Gotchas (what bit them)

Top-down evals can miss real-world usage patterns.

Root cause: Researchers choose harms or capabilities to test before knowing which use cases are actually common.

Resolution: Use Clio to ground evaluation priorities in aggregate real-world usage clusters.

Usage analytics creates a privacy-versus-insight tension.

Root cause: More direct access to conversations gives more insight but can feel invasive or ethically dubious.

Resolution: Use high-level summaries, aggregate clusters, privacy auditing, and minimum-size thresholds.

Cluster descriptions can accidentally leak unnecessary private details.

Root cause: A cluster name or description could include specific website names or other identifiers even after summarization.

Resolution: Instruct Claude to avoid private details and audit clusters for identifying information.

Pre-deployment testing alone is insufficient for new capabilities.

Root cause: Real users can creatively use features such as computer use in ways pre-deployment teams did not anticipate.

Resolution: Use Clio for post-deployment monitoring alongside proactive safety work.

Evaluation

Anthropic evaluated Clio through a privacy evaluation for its auditor and synthetic corpora with known ground-truth distributions, then checked whether Clio reconstructed those distributions across random and synthetic concerning data.

Metrics: privacy-auditor performance on privacy-preserving and synthetic non-privacy-preserving examples · ground-truth distribution reconstruction · synthetic corpus with tens of thousands of conversations · known category proportions such as 10% math, 5% coding, and 2% teddy-bear questions

Outcomes

• Clio lets Anthropic understand aggregate Claude use cases without humans reading raw conversations.
• The team used Clio to design more representative evaluations grounded in empirical usage.
• Clio helped identify language and cultural usage differences, safety-relevant clusters, suspicious activity, and over-refusal topics.
• Clio complements pre-deployment testing with post-deployment monitoring for capabilities such as computer use.

Clio operates on random samples of Claude conversations, enforces roughly-thousand-person privacy thresholds, validates against tens of thousands of synthetic conversations, and exposes aggregate clusters only after minimum organization and conversation thresholds.