Empowering Non-Technical Domain Experts to Drive AI Quality in Conversational AI

portola · Media & Entertainment · 2025 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Conversational AI quality operations

Portola needed to improve Tolan's emotionally complex AI-companion conversations when quality depended on memory authenticity, tone, character consistency, and subjective user experience that automated evals could not capture alone.

Constraints:

Conversation quality is subjective and context-dependent.
Tolan combines memory retrieval, user context, real-time voice processing, and user-shared content.
Non-technical experts own much of the relevant psychology, storytelling, and conversation-design judgment.
Prompts and models change quickly, making static golden datasets stale.
The source is a Braintrust customer story and has promotional bias.

Architecture

Portola built a human-centered quality workflow around Tolan's production traces. Subject matter experts review Braintrust logs, identify recurring behavioral issues, curate problem-specific datasets, compare prompt variants in playgrounds, and deploy approved prompt changes through prompts-as-code without engineering handoffs. The loop keeps expert judgment close to production behavior while preserving trace context for targeted iteration.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  tolan-production-conversations["Tolan production conversations"] --> braintrust-traces["Braintrust traces"]
  braintrust-traces --> subject-matter-experts["Subject matter experts"]
  subject-matter-experts --> problem-specific-datasets["Problem-specific datasets"]
  problem-specific-datasets --> playground-comparisons["Playground comparisons"]
  playground-comparisons --> prompts-as-code-deployment["Prompts-as-code deployment"]
  prompts-as-code-deployment --> tolan-production-conversations

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Tolan production conversations`	AI companion conversations whose logs expose memory, tone, and behavior issues.	`Braintrust traces`
`Braintrust traces`	Stores full conversation context and examples for review and dataset curation.	`Subject matter experts`
`Subject matter experts`	Behavioral researcher, writer, and designer who judge conversation quality and identify recurring issues.	`Problem-specific datasets`
`Problem-specific datasets`	Tagged collections of real examples for focused prompt iteration.	`Playground comparisons`
`Playground comparisons`	Side-by-side workspace for testing current and candidate prompts with expert review.	`Prompts-as-code deployment`
`Prompts-as-code deployment`	Allows approved prompt changes to ship to production without engineering handoffs.	`Tolan production conversations`

Data flow

Production conversations are logged in Braintrust. A domain expert reviews traces for recurring quality issues, curates tagged datasets of examples, runs side-by-side playground comparisons between current and candidate prompts, and ships approved prompt changes directly to production.

“Portola built a workflow that enables subject matter experts to identify issues, curate datasets, test solutions, and deploy changes-- all without engineering handoffs.”article, stated

“Once a dataset is curated, Lily moves to playgrounds for side-by-side prompt comparison. She manually reviews outputs from the current prompt versus iterations”article, stated

Why this architecture

Portola chose an expert-in-the-loop workflow because emotional intelligence, authentic memory, and brand voice could not be reduced to automated scores. Problem-specific datasets and playground comparisons gave experts enough production context to iterate quickly while avoiding stale all-purpose test suites.

“Tolan focuses on creating genuine emotional connections through natural voice conversations and complex memory systems.”article, stated

“couldn't be captured by automated evals alone.”article, stated

Models used: Tolan conversational AI · Braintrust

Scale

Subject matter experts spend hours daily reviewing logs and shipping prompt improvements.
“spend hours daily reviewing logs, curating datasets, and shipping prompt improvements directly to production without engineering bottlenecks.”article, stated
Lily spends about one hour each day reviewing chat logs.
“Lily Doyle, their behavioral researcher, spends about an hour each day reading through chat logs in Braintrust, looking for patterns in conversation quality.”article, stated
Problem-specific datasets range from 10 to 200 examples.
“Portola creates problem-specific datasets in Braintrust ranging from 10 to 200 examples.”article, stated
The workflow produced 4x the number of weekly prompt iterations.
“domain experts identify issues, create datasets, test solutions, and ship changes, resulting in 4x the number of weekly prompt iterations.”article, stated

Alternatives considered (roads not taken)

Rely on automated evals for conversation quality.

Rejected because: The team found that subjective qualities such as asking the best question in context are not easily captured by automated metrics.

“Things like asking the best question to the user for the context they gave you—that's not super eval-able.”article, stated

Maintain a single golden dataset.

Rejected because: Models and prompts changed too quickly, so a static golden dataset would become stale.

“"It feels useless to come up with a golden dataset," Lily explains. "We're on a different model, we've changed the prompt eight times. Things change so fast."”article, stated

Route prompt improvements through engineers.

Rejected because: Direct deployment let domain experts complete the full issue-to-production cycle without engineering bottlenecks.

“Now, domain experts identify issues, create datasets, test solutions, and ship changes, resulting in 4x the number of weekly prompt iterations.”article, stated

Patterns

`human-in-the-loop-gate` (evaluation)

Non-technical experts manually review logs and prompt outputs because conversation quality is subjective.

“She manually reviews outputs from the current prompt versus iterations, assessing conversation quality through her domain expertise.”article, stated

`eval-driven-development` (evaluation)

Problem-specific datasets turn recurring conversation issues into focused review sets for prompt iteration.

“When Lily identifies a recurring problem through log reviews, user feedback, or focus group sessions, she creates a dataset in Braintrust tagged with the specific issue.”article, stated

`prompt-versioning` (prompting)

Prompts-as-code lets subject matter experts test and deploy prompt changes directly once playground review succeeds.

“The final piece of Portola's workflow is their prompts-as-code infrastructure, which enables subject matter experts to deploy changes directly to production”article, stated

`agent-memory` (memory)

Tolan's quality bar includes memory behavior that feels like a friend rather than perfect recall.

“The memory system needs to work in a way that feels authentic to how a friend would remember things.”article, stated

Tradeoffs

Evaluation approach

Chose Manual expert review for subjective conversation qualities. over Automated scoring as the primary quality signal..

Rationale: Automated scorers may miss whether a response creates emotional resonance in its specific context.

“An automated scorer might flag a response as too long, but miss that the length created emotional resonance in that specific context.”article, stated

Dataset strategy

Chose Many issue-specific datasets. over One golden dataset..

Rationale: Focused datasets stay fresh and let experts address newly observed behavioral issues immediately.

“Each dataset targets a specific behavioral pattern, making it easier to measure improvements”article, stated

Release workflow

Chose Let subject matter experts ship prompt changes directly. over Engineering-mediated prompt deployments..

Rationale: Direct deployment increased iteration velocity and kept domain judgment in the loop end to end.

“This end-to-end autonomy transformed Portola's iteration velocity.”article, stated

Gotchas (what bit them)

Automated evals miss subjective conversational quality.

Root cause: The product depends on emotional intelligence, authentic behavior, and context-specific judgment.

Resolution: Use expert manual review in playgrounds and logs for squishy quality dimensions.

“"A lot of what we're working on is really squishy stuff," Lily explains.”article, stated

A comprehensive golden dataset would become stale.

Root cause: Prompts and models changed quickly, making a single reference suite mismatched to current product behavior.

Resolution: Create focused, problem-specific datasets for current issues.

“Datasets reflect current product state rather than becoming stale snapshots”article, stated

Authentic memory cannot be reduced to perfect recall.

Root cause: Companion conversations require nuanced decisions about what to remember and when it should surface.

Resolution: Review memory behavior qualitatively against the feeling of natural friendship.

“Perfect recall matters less than the subtle nuance of what gets remembered, when it surfaces naturally, and how it integrates into conversations.”article, stated

Evaluation

Portola evaluates prompt changes through expert log review, issue-specific datasets, and side-by-side playground comparisons rather than relying only on automated metrics.

Metrics: 4x weekly prompt iterations · Problem-specific datasets of 10 to 200 examples · About one hour per day of log review by a behavioral researcher

“Playgrounds serve as Lily's primary workspace, where she: * Loads datasets directly from curated logs * Runs comparison tests between prompt versions”article, stated