Building Reliable AI Agents Through Production Monitoring and Intent Discovery

raindrop · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production monitoring for conversational AI products

Raindrop addresses the reliability gap that appears when long-running AI agents move from offline evaluation into production, where users create unanticipated intents and failures often do not throw software errors.

Constraints:

• Offline input-output evals cannot cover long conversations, memory, tools, and user-specific context.
• AI failures often appear as bad behavior, refunds, refusals, wrong dates, or forgotten context rather than exceptions.
• Raw event counts can overstate impact when one frustrated user repeats the same complaint many times.
• Discovery can tolerate noisy clusters, but tracking and fix verification require more precise issue definitions.
• Classifier coverage must scale across production data without forcing expensive frontier-model judging on every event.
• Teams need to improve targeted workflows without destabilizing prompts and behaviors that already work.

Architecture

Raindrop frames AI-product reliability as a discover-track-fix loop over production conversations. The system ingests production user and assistant interactions, captures explicit feedback events such as thumbs-down, regenerations, copies, and relevance marks, and detects implicit signals such as refusals, task failures, forgetting, wrong dates, and user frustration. It clusters signals and intents to discover concrete issues, uses semantic search plus LLM-assisted binary matching to calibrate issue definitions, trains smaller custom models for broad production coverage, and reports issue impact by affected-user percentage and metadata such as model, modality, intent route, tools invoked, and tool errors. The fix loop encourages teams to convert repeated intents or issues into modular tool or sub-agent workflows so they can experiment and improve one bucket without rewriting the entire main prompt.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production-conversation-data[Production conversation data] --> explicit-feedback-signals[Explicit feedback signals]
  production-conversation-data --> implicit-signal-detectors[Implicit signal detectors]
  production-conversation-data --> intent-clustering[Intent clustering]
  explicit-feedback-signals --> issue-discovery-clusters[Issue discovery clusters]
  implicit-signal-detectors --> issue-discovery-clusters
  intent-clustering --> issue-discovery-clusters
  issue-discovery-clusters --> deep-search-calibrator[Deep search calibrator]
  deep-search-calibrator --> custom-issue-models[Custom issue models]
  custom-issue-models --> monitoring-dashboard[Monitoring dashboard]
  intent-clustering --> metadata-breakdown[Metadata breakdown]
  metadata-breakdown --> monitoring-dashboard
  monitoring-dashboard --> tool-subagent-layer[Tool subagent layer]
  metadata-breakdown --> tool-subagent-layer

Components

Data flow

Production conversations produce explicit signals from user actions and implicit signals from conversational behavior. Raindrop filters and clusters those signals with user intents to discover issue categories, calibrates candidate definitions through deep search and user labeling, trains smaller detectors for broad production coverage, and reports issue impact with affected-user percentages plus metadata such as model, modality, intent route, tools invoked, and tool errors. Teams then use the monitored issues to build evals, guardrails, prompt changes, or modular tool/sub-agent workflows and watch production alerts to see whether a change helped or hurt.

Why this architecture

Raindrop chooses production monitoring because long-running agents depend on conversation history, memory, tools, and real user context that offline eval sets cannot anticipate. It combines signals with intents because broad symptoms like user frustration are not actionable until they are tied to what the user was trying to do. It trains custom models rather than judging every production event with a frontier model because useful monitoring needs coverage across production data, not a tiny sampled slice. The Trellis/tool-subagent guidance exists because targeted workflows reduce prompt crosstalk and make experiments safer.

Models used: LLMs for semantic reranking and binary event matching · Custom small issue-detection models · Embedding or semantic search models · Specialized tool/sub-agent models such as the query-language model example

Scale

• The talk frames useful long-running agents as running for 15 to 20 minutes.
  “It was like the first time where you'd have something that would run for like 15 20 minutes and then like you're like, "Wow, that was worth 15 or 20 minutes, right?"”transcript, stated
• Production impact is prioritized by affected-user share, with 80% of users used as the example of an urgent issue.
  “if you saw that it happened to 80% of your users, you're going to like wake up, you know, your engineers”transcript, stated
• Raindrop's detectors are designed for millions of production events per day.
  “We train a little model under the hood so that we can process millions of events a day without anyone going broke.”transcript, stated
• The Leave customer example involved a studio with five-plus million users.
  “one of our customers leave which has they're a studio that has like five plus million users”transcript, stated
• The deep-search example turns labeled examples into a detector for issues affecting 8% of users.
  “this issue that you've defined and you've kind of calibrated what it means um is you know present is affecting 8% of users”transcript, stated
• Tool failure monitoring explicitly considers repeated failures such as five or ten consecutive tool errors.
  “if you see that a tool failed like five or 10 times in a row, um that's that's very interesting, right?”transcript, stated

Alternatives considered (roads not taken)

Rely mainly on offline eval input-output pairs

Rejected because: Long-running agents accumulate context, memory, tools, and external state that make fixed input-output expectations insufficient.

Monitor only traditional latency, token, and cost traces

Rejected because: Raindrop wants product-quality monitoring that asks whether the app is doing well and whether a prompt or model is better in the real world.

Use subjective LLM-as-judge scores for every production output

Rejected because: Scored judging requires its own eval product and cannot affordably cover enough production data.

Count raw signal events

Rejected because: Raw counts can be dominated by one user repeating the same problem, so user-level percentages better represent impact.

Put all domain and query-language knowledge in the main agent prompt

Rejected because: The main agent became slower and confused when the query language lived in the central prompt, so Raindrop used a separate model invoked as a tool.

Patterns

observability-tracing (monitoring)

Raindrop treats production conversations as the source of reliability truth, tracking live signals, affected users, metadata, and alerts rather than only offline test outcomes.

eval-driven-development (quality)

Offline evals remain part of the reliability stack, but Raindrop uses production discovery to decide which issues deserve evals, guardrails, prompt changes, or monitoring.

data-quality-loop (feedback)

The system collects explicit and implicit feedback signals, clusters them into issues, tracks impact, and uses those findings to refine issue definitions and product behavior.

llm-as-judge (evaluation)

Raindrop uses LLMs for constrained binary event matching inside deep search, but distinguishes that from open-ended subjective scoring.

model-distillation (modeling)

After users calibrate an issue definition, Raindrop trains a smaller model to apply that detector across production traffic economically.

tool-use-orchestration (architecture)

The talk recommends treating tools as sub-agents and splitting domain-specific functionality into modular workflows that can be monitored and improved independently.

Tradeoffs

Combine offline evals with production monitoring

Chose Offline evals plus live production signals over Treating eval suites as sufficient reliability coverage.

Rationale: Offline evals are useful, but real users expose intents and failure modes that builders cannot predict before deployment.

Detect specific issues instead of judging open-ended quality

Chose Binary issue detection over subjective scoring over Asking an LLM to rate output quality or writing quality.

Rationale: Specific issue presence is easier to calibrate and scale than broad, subjective score judgments.

Measure affected users instead of raw events

Chose User-percentage signal tracking over Raw signal-event counts.

Rationale: Affected-user percentages better represent business impact and avoid over-weighting repeated complaints from one user.

Train custom detectors for broad coverage

Chose Calibrated custom issue models over Using the same or larger model to judge every production output.

Rationale: Production monitoring is most useful when it covers all data, and frontier-model judging at that scale is too expensive.

Decompose product capabilities into tool or sub-agent buckets

Chose Modular tools and specialized sub-agents over One general agent prompt that owns every capability.

Rationale: Buckets reduce crosstalk, allow targeted model experiments, and make changes safer when existing behavior already works.

Sometimes fix with guardrails before writing large eval suites

Chose Direct fixes or guardrails for obvious bugs over Always expanding eval coverage first.

Rationale: The Q&A notes that some known issues are better fixed directly rather than spending time inventing difficult evals around them.

Gotchas (what bit them)

Offline eval sets miss failures caused by long conversation state.

Root cause: Agents accumulate memory, tools, custom tool sets, and context over days or weeks, so correctness depends on state that fixed input-output pairs cannot represent.

Resolution: Use offline evals where useful, but add production monitoring and issue discovery from live conversations.

AI product failures can be invisible to traditional error monitoring.

Root cause: Bad agent behavior often appears as wrong promises, refusals, forgotten context, or unsafe conversations rather than thrown exceptions.

Resolution: Detect explicit and implicit conversational signals instead of relying only on exceptions or spans.

Raw event counts can misrepresent how widespread a problem is.

Root cause: A single user can generate many complaint events, making event counts look large without indicating broad user impact.

Resolution: Track signal changes by percentage of affected users per day.

Discovery-quality clusters are not precise enough for fix verification.

Root cause: Loose clustering is useful for finding unknown issues, but tracking requires accurate definitions so teams know whether a fix worked.

Resolution: Refine issue definitions, calibrate search examples, and train detectors before using them for tracking.

Subjective LLM-as-judge scoring becomes a second product to maintain.

Root cause: Teams need their judge outputs to match human labels, so every scored criterion requires its own eval and prompt-calibration work.

Resolution: Prefer calibrated binary classifiers for specific issues, with humans marking examples and detectors learning the definition.

Adding domain knowledge to one large agent prompt can slow and confuse the product.

Root cause: The main agent may start applying specialized query-language context everywhere, and prompt changes become coupled to every product behavior.

Resolution: Move specialized capabilities into tool-like sub-agents that receive a natural-language task description.

Evaluation

The evaluation approach is production-first: define explicit and implicit signals, cluster them into issues and intents, track affected-user percentages and metadata, calibrate binary detectors through deep search and labeled examples, and keep offline evals for critical or repeatable cases.

Metrics: Percentage of users hitting a signal each day · Event count and number of affected users · Thumbs up, thumbs down, regeneration, copy, upgrade, and relevance-mark feedback · Implicit issue presence such as refusals, task failures, forgetting, wrong dates, or user frustration · Model, modality, intent, tool invocation, tool-error, and average tool-call metadata · Issue prevalence such as an 8% affected-user detector · Latency, token counts, and costs as complementary operational metrics

Outcomes

• Raindrop articulated a discover-track-fix loop for AI-product reliability.
• The platform trains smaller detectors so issue definitions can run over millions of events per day.
• Deep search turns semantic search, LLM reranking, and human labels into calibrated issue monitors.
• The monitoring approach complements tracing platforms such as LangSmith or Braintrust rather than replacing them.
• The talk reports no controlled benchmark lift or customer-specific reliability improvement.

Concrete operating scale includes long-running agents of 15 to 20 minutes, detectors intended for millions of production events per day, issue monitors that can report examples like 8% affected users, and customer context at five-plus million users. The talk does not provide latency, cost, precision/recall, or before/after reliability metrics for Raindrop itself.