Building an AI Interview Coach for Product Discovery Training

product_talk · Education · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Product discovery training feedback

Product Talk needed to give students useful feedback on practice customer interviews at course scale without mis-teaching story-based interviewing or mishandling real customer transcript data.

Constraints:

• Students need deliberate practice with feedback to learn interviewing skills.
• AI feedback that is slightly wrong can derail learners.
• The course rubric spans multiple dimensions that can conflict if evaluated independently.
• Real customer interview transcripts require stronger privacy and compliance handling than practice transcripts.

Architecture

The Interview Coach evolved from custom GPT/Claude prototypes into a production workflow. Students submit transcripts through LearnWorlds; consent and privacy filtering gate processing; Airtable stores traces; Anthropic API calls evaluate rubric dimensions; the system emails feedback to students and Teresa; error analysis turns trace annotations into code-based and LLM-as-judge evals; and the production path is moving from Zapier to AWS Lambda and Step Functions with Vistaly for SOC 2-compliant real interview transcripts.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  student[Student] --> learnworlds-course-platform[LearnWorlds Course Platform]
  learnworlds-course-platform --> zapier-workflow[Zapier Workflow]
  zapier-workflow --> airtable-trace-store[Airtable Trace Store]
  zapier-workflow --> anthropic-api[Anthropic API]
  zapier-workflow --> email-feedback[Email Feedback]
  email-feedback --> student[Student]
  email-feedback --> human-reviewer[Human Reviewer]
  human-reviewer --> airtable-trace-store[Airtable Trace Store]
  airtable-trace-store --> eval-notebook[Eval Notebook]
  eval-notebook --> aws-lambda[AWS Lambda]
  learnworlds-course-platform --> aws-step-functions[AWS Step Functions]
  vistaly[Vistaly] --> aws-step-functions[AWS Step Functions]
  aws-step-functions --> aws-lambda[AWS Lambda]
  aws-lambda --> anthropic-api[Anthropic API]

Components

Data flow

A student submits a transcript and consent through the course platform; a Zap or later Lambda/Step Functions workflow stores the trace, runs separate LLM calls for rubric dimensions, formats JSON into feedback with excerpts and tips, emails the student and Teresa, runs evals against the resulting trace, and feeds errors back into prompt/model/temperature changes.

Why this architecture

The system evolved as real usage exposed failure modes. Chat interfaces were abandoned because students wanted multi-turn coaching that the tool was not designed for; one long prompt was split into rubric-dimension calls; traces became the evaluation substrate; and AWS/Vistaly replaced Zapier/Airtable where reliability and SOC 2 handling mattered.

Models used: ChatGPT · Claude · Anthropic API · GPT-4o mini

Scale

• The first prototype required only $20 ChatGPT and $20 Claude subscriptions.
  “All that was required for this first MVP was a $20 subscription to Chat GPT and a $20 subscription to Claude. I tried out both. I wrote a really simple prompt. In fact, if you want to steal this, you can because it relies entirely on public”transcript, stated
• The Replit tool was built in two days.
  “And let me tell you, I had never used Replit before. I built this tool in two days. It blew my mind. I was like, okay, wow, the future really is here. Uh this is spectacular.”transcript, stated
• The early rate limit was two practice interviews per week.
  “limit artificially. So, by design I set the rate limit as two practice interviews a week. That's because they do a practice interview in class. I wanted them to be able to run their their transcript through the interview coach. And then if they tried it in”transcript, stated
• Students consented to storing traces for 90 days.
  “then they have to give me permission uh to store the trace for 90 days. Once they submit this assignment, it triggers a zap. You can see this is triggered by a LearnWorlds assessment.”transcript, stated
• The coach eventually measured seven dimensions and about a dozen evals for one dimension.
  “I have seven dimensions to the interview coach. This is one dimension. I actually have about a dozen evals for this one dimension.”transcript, stated
• The production partnership targeted a July beta and larger September launch.
  “we are hoping to launch um a beta group in late July and um a much bigger launch in September. So especially for those of you who are listening, if you're in if you're a Vistaly customer and you're able to um join our July cohort of continuous”transcript, stated

Alternatives considered (roads not taken)

Keep the custom GPT/Claude prototype.

Rejected because: It was useful for proving feedback quality but not a deployable course tool.

Keep the Replit chat-interface builder.

Rejected because: Students used chat turns for follow-up questions, rate limits broke, and the tool became hard to extend reliably.

Use one long prompt for the whole coach.

Rejected because: The model confused rubric criteria as the prompt grew longer.

Split every rubric dimension into independent prompts without cross-dimension context.

Rejected because: The split improved excerpts but caused inconsistent feedback when one dimension judged another's concern.

Use Zapier/Airtable for real customer transcripts long term.

Rejected because: The setup was not SOC 2 compliant and not intended for real customer interview data.

Patterns

eval-driven-development (quality)

Trace error analysis drives code-based and LLM-based evals that decide whether prompt, model, temperature, or chunking changes improve the coach.

llm-as-judge (quality)

LLM judges detect qualitative failures such as leading-question recommendations in coach responses.

prompt-chaining (workflow)

The coach split rubric dimensions into separate LLM calls and then formatted their JSON outputs into one response.

observability-tracing (operations)

The system stores user inputs and LLM responses as traces for monitoring, annotation, evals, and release decisions.

human-in-the-loop-gate (quality)

Teresa reviewed outputs, corrected students when necessary, and annotated traces before broader automation.

privacy-preserving-processing (data-governance)

The workflow added consent, sensitive-data removal instructions, 90-day trace storage, and Vistaly for SOC 2 handling.

durable-execution (workflow)

The production version moves orchestration to AWS Step Functions so Node.js and Python steps can run reliably.

Tradeoffs

Prototype speed

Chose ChatGPT/Claude and Replit first over Starting with production infrastructure.

Rationale: Cheap prototypes quickly proved whether LLM feedback could model the rubric.

Student-facing form factor

Chose Assignment submission with emailed feedback over Embeddable chat interface.

Rationale: The intended job was one feedback pass over a transcript, not a multi-turn coaching conversation.

Prompt architecture

Chose Multiple rubric-dimension calls over One monolithic prompt.

Rationale: Splitting prompts enabled excerpts and better dimension-specific feedback, though it required new evals for consistency.

Eval implementation

Chose Own Jupyter/Python evals first over An off-the-shelf eval tool.

Rationale: Torres wanted to learn the process and avoid adopting a tool's point of view before understanding evals.

Production orchestration

Chose AWS Lambda and Step Functions over Zapier automation.

Rationale: The production version needed a more reliable environment that could orchestrate Node.js and Python.

Gotchas (what bit them)

A chat interface changed the intended product behavior.

Root cause: Students treated feedback as a multi-turn coaching conversation, consuming message-based rate limits.

Resolution: Move from chat to transcript submission plus one structured feedback response.

Long prompts blurred rubric criteria.

Root cause: One prompt carried context for all dimensions and grew unwieldy as errors were patched.

Resolution: Split dimensions into separate calls, then add evals to catch cross-dimension inconsistencies.

Splitting prompts improved local quality but introduced inconsistency.

Root cause: Each dimension-specific prompt lost awareness of what another dimension was supposed to cover.

Resolution: Use trace review and evals to catch inconsistent cross-dimension feedback.

Vibe-checking AI feedback hid serious errors.

Root cause: Student praise and manual trace inspection did not reveal systematic bad recommendations.

Resolution: Annotate traces, identify error categories, and build code/LLM evals for those failures.

Practice-transcript infrastructure was not safe for real customer data.

Root cause: Students uploaded real interviews even though transcripts were stored in Airtable without SOC 2 compliance.

Resolution: Add consent and sensitive-data instructions, then partner with Vistaly for SOC 2-compliant real transcript handling.

Evaluation

Product Talk evaluates the coach through trace review, Airtable annotations, error analysis, code-based evals, LLM-as-judge evals, judge scoring against human grades, and repeated experiments over model, prompt, temperature, and chunking changes.

Metrics: Pass/fail by error category · Human grader agreement · Judge false positives · Judge false negatives · Code-eval failures · Student feedback usefulness · Release/toss decision by eval grid

Outcomes

• The day-one GPT/Claude prototype correctly identified non-story-based interview behavior.
• A Replit version let the team deploy embedded teaching tools in two days.
• Real student usage exposed rate-limit, trace, privacy, rubric-confusion, and consistency failures.
• Error-analysis-driven evals gave a fast feedback loop for releasing or rejecting changes.
• The production path moved to AWS Lambda/Step Functions and Vistaly for real customer transcript support.

The talk cites a one-hour/day-one prototype, two-day Replit build, two practice interviews per week, 90-day trace storage, seven coach dimensions, about a dozen evals for one dimension, three days to build initial Python evals, a late-July beta, and a larger September launch.