Multi-Path AI Evaluation Framework for Marketplace Trust and Safety

thumbtack · Tech · 2026 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Marketplace GenAI evaluation

Thumbtack needed a scalable evaluation framework for GenAI features where tone, accuracy, usefulness, groundedness, and safety directly affect marketplace trust.

Constraints:

GenAI outputs are probabilistic and can be inconsistent, inaccurate, or harmful.
Product teams need rapid experimentation without duplicating evaluation infrastructure.
Marketplace trust requires safety and quality checks across customer and pro experiences.
Different use cases need different evaluation depth and cadence.

Architecture

Thumbtack built a hybrid, multi-path evaluation framework combining representative datasets, operationalized rubrics, rule checks, AI-as-a-judge, safety reviews, crowdsourced validation, MLflow rapid experiments, Databricks nightly jobs, and high-volume content pipelines.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  evals-team["Evals Team"]
  product-teams["Product Teams"]
  mlflow-evaluation-path["MLflow Evaluation Path"]
  databricks-evaluation-path["Databricks Evaluation Path"]
  high-volume-pipeline["High-Volume Pipeline"]
  quality-rubrics["Quality Rubrics"]
  human-validation["Human Validation"]
  ai-as-a-judge["AI-as-a-Judge"]
  rule-based-checks["Rule-Based Checks"]
  evaluation-paths["Evaluation Paths"]
  evals-team --> product-teams
  evals-team --> evaluation-paths
  product-teams --> evals-team
  product-teams --> evaluation-paths
  mlflow-evaluation-path --> product-teams
  mlflow-evaluation-path --> quality-rubrics
  databricks-evaluation-path --> product-teams
  databricks-evaluation-path --> quality-rubrics
  high-volume-pipeline --> rule-based-checks
  high-volume-pipeline --> ai-as-a-judge
  high-volume-pipeline --> human-validation
  quality-rubrics --> mlflow-evaluation-path
  quality-rubrics --> databricks-evaluation-path
  quality-rubrics --> ai-as-a-judge
  human-validation --> high-volume-pipeline
  ai-as-a-judge --> quality-rubrics
  ai-as-a-judge --> high-volume-pipeline
  rule-based-checks --> high-volume-pipeline

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Evals Team`	Provides shared evaluation tooling, content guidelines, and human oversight infrastructure.	`Product Teams`, `Evaluation Paths`
`Product Teams`	Own feature-specific iteration and prompt/model development.	`Evals Team`, `Evaluation Paths`
`MLflow Evaluation Path`	Supports fast-track evaluation, traces, rubric scoring, and prompt/model comparisons.	`Product Teams`, `Quality Rubrics`
`Databricks Evaluation Path`	Runs nightly jobs and deep analysis.	`Product Teams`, `Quality Rubrics`
`High-Volume Pipeline`	Evaluates high-volume generated content through layered checks.	`Rule-Based Checks`, `AI-as-a-Judge`, `Human Validation`
`Quality Rubrics`	Operationalize clarity, tone, groundedness, usefulness, and safety criteria.	`MLflow Evaluation Path`, `Databricks Evaluation Path`, `AI-as-a-Judge`
`Human Validation`	Crowdsourced or expert review of representative samples.	`High-Volume Pipeline`
`AI-as-a-Judge`	Scores semantic quality criteria at scale.	`Quality Rubrics`, `High-Volume Pipeline`
`Rule-Based Checks`	Validate structure, formatting, length, and schema compliance.	`High-Volume Pipeline`

Data flow

Product teams run evaluations through one of three paths: MLflow for rapid prompt/model experimentation, Databricks for nightly deep analysis, or a layered high-volume generation pipeline. Shared rubrics, judge models, safety review, and human validation provide common quality signals.

“The solution features three parallel evaluation paths supporting different team workflows: MLflow-based rapid experimentation, Databricks nightly jobs for deep analysis, and a multi-layer pipeline for high-volume content generation.”summary, stated

“This multi-method approach reflects a mature understanding that no single evaluation technique suffices for production GenAI systems.”summary, stated

Why this architecture

Thumbtack avoided a single rigid evaluator because teams vary in speed, depth, and data needs. Central evaluation standards reduce duplicated effort while product teams keep iteration ownership.

“This organizational model represents a thoughtful balance between centralized standards and distributed execution.”summary, stated

“different teams have varying needs regarding development speed, data complexity, and quality requirements.”summary, stated

Models used: LLM judge models · DeepEval GEval · MLflow GenAI stack

Scale

The framework supports GenAI across search, project summaries, pro listings, and marketing content.
“enabled Thumbtack to deploy AI across search, project summaries, pro listings, and marketing content”summary, stated
The Evals team spans applied science, ML infrastructure, content design, and trust and safety.
“a dedicated cross-functional Evals team comprising applied science, ML infrastructure, content design, and trust & safety expertise.”summary, stated

Alternatives considered (roads not taken)

Decentralized team-specific evaluations only

Rejected because: It caused duplicated effort and siloed learnings as AI features proliferated.

“the proliferation of AI features across the company led to duplicated effort and siloed learnings.”summary, stated

One rigid evaluation framework

Rejected because: Different workflows require different speed, data complexity, and quality depth.

“A distinctive aspect of Thumbtack's framework is the provision of three parallel evaluation paths, each optimized for different workflows and use cases.”summary, stated

Patterns

`eval-driven-development` (quality)

Evaluation paths and rubrics guide prompt/model iteration across teams.

“Thumbtack's evaluation approach rests on three foundational pillars: representative data, operationalized quality rubrics, and hybrid evaluation methods.”summary, stated

`llm-as-judge` (quality)

AI-as-a-judge scores semantic properties such as clarity, tone, relevance, groundedness, and accuracy.

“AI-as-a-judge evaluations use large language models to assess semantic properties like clarity, tone, accuracy, relevance, and groundedness”summary, stated

`human-in-the-loop-gate` (governance)

Crowdsourced and expert human review validates representative AI-approved content.

“crowdsourced human review via Amazon Mechanical Turk validates representative samples of AI-approved content during both pre-production and production phases.”summary, stated

Tradeoffs

Centralize standards, distribute iteration

Chose Cross-functional Evals team plus product-team ownership over Either fully centralized or fully decentralized evaluation.

Rationale: This balances shared infrastructure with domain-specific experimentation.

“product teams retained ownership of their iteration cycles while the central team provided shared tooling, content guidelines, and human oversight infrastructure.”summary, stated

Support three evaluation paths

Chose MLflow, Databricks nightly analysis, and high-volume pipeline over One universal evaluation workflow.

Rationale: Different teams need different setup effort, depth, and throughput.

“each optimized for different workflows and use cases.”summary, stated

Gotchas (what bit them)

Evaluation patterns duplicated across product teams.

Root cause: Early decentralization let teams solve similar evaluation problems independently.

Resolution: Create a cross-functional Evals team with shared tooling while keeping team iteration ownership.

“The dedicated Evals team provides continuity and shared infrastructure, preventing the reinvention of evaluation patterns across teams”summary, stated

Subjective guidelines need operational rubrics.

Root cause: Tone, usefulness, groundedness, and safety are hard to evaluate consistently as prose guidelines.

Resolution: Convert design, linguistic, and safety expectations into concrete criteria.

“Quality rubrics operationalize design, linguistic, and safety guidelines into concrete evaluation criteria.”summary, stated

Evaluation

Thumbtack combines rule checks, AI-as-a-judge, safety review, crowdsourced human validation, representative datasets, and rubrics across three workflow paths.

Metrics: clarity · tone · groundedness · usefulness · safety · schema compliance

“combining rule-based checks, AI-as-a-judge scoring, safety reviews, and crowdsourced human validation.”summary, stated

Outcomes

Thumbtack deployed GenAI across search, summaries, pro listings, and marketing content.
A central Evals team reduced duplicated evaluation effort.
Three evaluation paths support rapid experimentation, deep analysis, and high-volume content generation.

The case spans marketplace features across customer and professional touchpoints, but does not disclose traffic volume.

“providing pro recommendations, and creating marketing content at scale.”summary, stated