Building an AI-Powered Interview Coach with Comprehensive Evaluation Framework

product_talk · Education · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Educational coaching and LLM evaluation

Product Talk needed an AI interview coach that could give students useful feedback on customer-interview transcripts while building enough evaluation confidence to deploy the coach to real course students.

Constraints:

• Students need expert feedback, but peers are not yet experts in interviewing.
• The coach must grade a specific four-dimension rubric, not generic interviewing quality.
• Interview transcripts are large, making evaluation and repeated runs expensive.
• The builder needed evaluation code that could move from notebooks into production guardrails.

Architecture

Students upload SRT interview transcripts as homework. The interview coach analyzes the transcript and sends an email with scores, excerpts, and tips across four dimensions: story-based opening question, setting the scene, building the timeline, and redirecting generalizations. The product was built alongside an evaluation stack containing human labels, code assertions, and LLM-as-judge evals, with production traces compared against human judgment and eval code moved from notebooks into the production repository.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  course-platform[Course Platform] --> interview-coach[Interview Coach]
  interview-coach --> feedback-email[Feedback Email]
  interview-coach --> eval-suite[Eval Suite]
  human-labels[Human Labels] --> eval-suite[Eval Suite]
  eval-suite --> analysis-notebook[Analysis Notebook]
  production-repository[Production Repository] --> eval-suite[Eval Suite]

Components

Data flow

A student uploads an SRT transcript to the course platform. The coach produces a detailed feedback email containing rubric scores, transcript excerpts, and coaching tips. Production traces are annotated by humans, evaluated by code and LLM judges, visualized in notebooks, and eventually run from the development and production codebase as guardrails.

Why this architecture

The coach architecture follows the pedagogy: students practice, receive rubric-specific feedback, and improve through deliberate practice. The eval architecture exists because model, prompt, and temperature changes require a fast feedback loop grounded in human labels before release.

Models used: LLM-as-judge evaluators · AI interview coach model

Scale

• The speaker had been building the AI product for four months.
  “what I'm building. I have been for the last four months building an AI product. We'll get into the overview of that in just a little bit. I am forcing myself on a weekly basis to work out loud. I'm doing this one as a way to build in a little bit of accountability to keep myself sort of focused.”transcript, stated
• The coach gives feedback across four rubric dimensions.
  “is that this instance of the interview coach gives feedback on four different dimensions. So, in the course, we teach collecting a story by teaching opening with a story-based question. That's dimension one. Setting the scene, that's dimension two. Building with a timeline.”transcript, stated
• The development set was 25 transcripts, with a larger test set used before release.
  “running evals is expensive. So my dev set is only 25 transcripts. My test set is much bigger. Um, but while I'm developing I work on a small set and then before I release I run on a large test set to make sure what I found in development holds on the larger set. So”transcript, stated
• The course context includes cohorts of roughly 20 to 50 students.
  “courses where we have 20 to 50 students in a cohort. So I'm not looking at thousands and thousands of traces. I actually run my evals on every trace that's coming in. We're about to release”transcript, stated

Alternatives considered (roads not taken)

Rely on peer feedback alone.

Rejected because: Students are practicing but are not yet expert interviewers.

Keep evals only in notebooks.

Rejected because: Production guardrails require eval code in the repo and deployed environments.

Depend on one type of evaluation.

Rejected because: The system combines human evals, code assertions, and LLM judges because each catches different issues.

Patterns

eval-driven-development (evaluation)

Product Talk used evals to decide whether coach, prompt, model, or temperature changes improved the product before release.

llm-as-judge (evaluation)

The evaluation setup includes LLM judges that are compared against human labels.

human-in-the-loop-gate (quality-control)

The coach remains transparent about beta status and every outgoing email is reviewed during error analysis.

Tradeoffs

Evaluation workflow

Chose Small dev set plus larger pre-release test set over Run all transcripts through every eval during iteration.

Rationale: The small set keeps iteration practical while the larger set gives release confidence.

Eval implementation location

Chose Production repo for eval code and notebook for analysis over Duplicate eval code inside notebooks.

Rationale: This allows production guardrails without losing notebook-based visualization.

Beta transparency

Chose Tell students feedback is AI-generated and manually review emails over Present the coach as fully reliable automation.

Rationale: Known issues remained, and review was part of error analysis.

Gotchas (what bit them)

Eval failures can reveal product bugs rather than model errors.

Root cause: The coach's own response parsing accidentally included questions that were examples of good questions.

Resolution: Use eval dashboards to inspect failures and trace them back to core-system bugs.

Human intuition may be less precise than a well-specified deterministic eval.

Root cause: Writing the eval forced clearer rubric definitions than manual labeling.

Resolution: Compare eval output to human labels but also refine the rubric when the eval is more consistent.

Large transcripts make exhaustive eval runs expensive.

Root cause: Interview transcripts are long and repeated LLM evaluation is costly.

Resolution: Use a small dev set during development and a larger test set before release.

Evaluation

The project itself is evaluation-centered: human annotations create labels, code assertions catch deterministic errors, LLM judges evaluate qualitative failures, scores compare eval output against human labels, and production traces are rerun against evals before changes ship.

Metrics: Alignment with human labels · True positives and false positives · Eval failure counts · Problematic question categories · Dev-set and test-set performance · Production trace guardrail results

Outcomes

• The coach produced student feedback emails with rubric scores, excerpts, and tips.
• The evaluation work uncovered bugs in the core system and clarified the teaching rubric.
• Eval code moved from notebooks into the production repository so it could run as guardrails.

The source describes four months of work, four feedback dimensions, a 25-transcript dev set, larger test set, production traces, and course cohorts of 20-50 students.