Large-Scale Tax AI Assistant Implementation for TurboTax

intuit · Finance · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Regulated tax-assistant question answering and explanation

Intuit needed TurboTax users to understand deductions, credits, refunds, product steps, and personal tax situations at the scale of tens of millions of annual returns. The assistant had to route static and dynamic questions to the right LLM/RAG/tax-engine solution, keep calculations authoritative, satisfy legal/privacy/security constraints, handle slow LLM latency during tax-season peaks, and use tax experts to verify quality in a domain where hallucinated numbers can create financial and legal harm.

Constraints:

• TurboTax processed 44 million tax returns for tax year 2023
• the domain is regulated and safety/security are described as very important
• tax calculations and numbers must come from proprietary tax knowledge engines, not LLM arithmetic
• static tax explanations and dynamic user questions require different solutions
• model changes and annual tax-law changes require re-evaluation
• LLM latency is measured in seconds and can rise near April 15
• consented data and regulated environments are required for fine-tuning

Architecture

Intuit built Intuit Assist for TurboTax on GenOS, its generative-AI platform. GenUX handles the customer-facing experience, an orchestrator routes questions to the appropriate LLM solution, Claude powers the main static tax-explanation use case, GPT-4 Mini has handled dynamic Q&A, and RAG/Graph RAG integrate proprietary tax information and IRS changes. Tax calculations stay in Intuit's tax knowledge engine, while LLMs explain the results. Human tax analysts act as prompt engineers and evaluators, seed golden datasets, and support LLM-as-judge automation and production monitoring.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  workflow["TurboTax user workflow"] --> genux["GenUX"]
  genux --> orch["GenOS orchestrator"]
  orch --> static["Static explanation service"]
  orch --> dynamic["Dynamic Q&A service"]
  static --> claude["Claude model endpoint"]
  dynamic --> gpt["GPT model endpoint"]
  dynamic --> rag["RAG and Graph RAG layer"]
  rag --> tax["Tax knowledge engine"]
  tax --> static
  static --> guard["Safety guardrails"]
  dynamic --> guard
  experts["Tax expert evaluation team"] --> static
  experts --> golden["Golden dataset"]
  golden --> judge["LLM-as-judge evaluator"]
  gpt --> judge
  judge --> monitor["Production monitoring"]
  guard --> monitor
  fine["Fine-tuning environment"] --> claude

Components

Data flow

A TurboTax user enters tax information and may view a static explanation or ask a dynamic question. GenUX presents the assistant experience, while the GenOS orchestrator identifies which team-owned solution should answer the query. Static refund or deduction explanations use prepared prompts with tax profile data and Claude; dynamic questions use Q&A models such as GPT-4 Mini plus RAG or Graph RAG over tax knowledge. Numerical facts come from Intuit's tax engine. Human tax analysts design prompts, manually evaluate early outputs, create golden data through tools such as AWS Ground Truth, and their judgments become automated LLM-as-judge and monitoring checks for production interactions.

Why this architecture

Intuit uses platform orchestration because different tax, product, static, and dynamic questions need different solutions. The architecture separates deterministic tax calculations from LLM explanations so numerical truth remains in the tax engine. Human tax experts define the prompts and evaluation baseline because tax accuracy is domain-specific, while LLM-as-judge and production monitoring make repeated model, prompt, and tax-year changes scalable. Fine-tuning was tested to reduce prompt size and latency, but only within regulatory and consent constraints.

Models used: Claude · Claude 3 Haiku · Claude Instant · GPT-4 Mini · GPT-4 series · RAG · Graph RAG · ML safety models

Scale

• TurboTax processed 44 million tax returns for tax year 2023.
  “Turboax successfully processed 44 million tax returns for tax year 23”transcript, stated
• The Claude relationship was described as a multi-million dollar contract.
  “The production model that we went with is claude uh for this use case. Uh in is one of the uh biggest users of claude.”transcript, stated

  “Uh we had a multi-million dollar contract for this year as well.”transcript, stated
• LLM latency is described as 3, 5, or 10 seconds rather than 100-200 milliseconds.
  “We're not looking at, you know, 100 millisecond, 200 milliseconds. We're talking about 3 seconds, 5 seconds, 10 seconds.”transcript, stated
• Tax-day traffic around April 15 increases latency pressure.
  “as you go into you know tax day everybody's trying to file on tax day right April 15 so uh latency really is uh shooting through the roof.”transcript, stated
• Annual tax-year changes such as 2023 to 2024 require reevaluation.
  “the laws change in the IRS say tax year 23 to 24 what happened”transcript, stated

Alternatives considered (roads not taken)

Use one generic LLM solution for all TurboTax assistant questions

Rejected because: The speaker says different question types need different solutions and an orchestrator/planner routes to the right component.

Let the LLM calculate tax numbers

Rejected because: Intuit keeps numbers in its proprietary tax engine and uses guardrails to prevent hallucinated numbers in explanations.

Rely only on manual expert evaluation for every prompt tweak

Rejected because: Manual evaluation creates the baseline, then automated LLM-as-judge evaluation handles smaller iterations while major changes return to experts.

Treat model upgrades as simple provider-version swaps

Rejected because: Intuit found even same-vendor upgrades require evaluation because prompts, tax rules, and model behavior change.

Patterns

model-routing (orchestration)

GenOS routes static explanations, dynamic tax questions, product questions, RAG flows, and model endpoints to the appropriate Intuit solution.

rag-retrieval (retrieval)

TurboTax uses RAG and Graph RAG over tax information and proprietary knowledge engines to answer tax questions with current domain context.

human-in-the-loop-gate (evaluation)

Tax analysts provide prompts, manual evaluations, and golden examples before automation is trusted for ongoing changes.

llm-as-judge (evaluation)

Intuit uses LLM-as-judge prompts, golden data, and GPT-4-series models to scale quality checks after expert baselines are established.

fine-tuning-adaptation (model-adaptation)

Intuit tested Claude 3 Haiku fine-tuning on AWS Bedrock for static tax queries to reduce prompt size and preserve quality.

safety-policy-enforcement (governance)

Safety guardrails and ML checks prevent hallucinated tax numbers and keep regulated outputs tied to authorized tax-engine data.

Tradeoffs

Numerical authority

Chose tax-engine numbers with LLM-generated explanations over LLM-generated tax calculations.

Rationale: This keeps regulated numerical truth in a proven internal system while still giving users natural-language explanations.

Evaluation cadence

Chose manual expert evaluation for baselines and major changes, automated eval for smaller iterations over manual-only or automated-only evaluation.

Rationale: Expert judgment sets the tax-quality bar, while automated evaluation keeps prompt and model iteration practical.

Fine-tuning scope

Chose pilot Claude 3 Haiku fine-tuning for static queries over broadly fine-tuning every assistant behavior.

Rationale: Static explanations offered a bounded test case for reducing prompt size and latency, while dynamic questions remained model/RAG-driven.

Product latency strategy

Chose design fallbacks and UX around multi-second LLM latency over expecting LLMs to meet 100-200ms backend SLAs.

Rationale: Tax-season prompts and load make multi-second response times unavoidable, so product design must absorb the delay.

Gotchas (what bit them)

LLM tax assistants must not hallucinate numbers.

Root cause: Tax answers include refund, deduction, and tax-profile numbers where small errors can be consequential.

Resolution: Use the proprietary tax knowledge engine as the source of numbers and add ML safety checks over raw LLM responses.

Provider and prompt lock-in are operational, not just commercial.

Root cause: Long-term vendor contracts reduce cost, while prompts and same-vendor upgrades require rework and validation.

Resolution: Maintain strong vendor partnerships and a clear evaluation framework for model and prompt changes.

LLM latency does not behave like normal backend latency.

Root cause: Complex tax situations make prompts balloon, and peak filing periods increase load.

Resolution: Design the product experience, fallback mechanisms, and user interface around multi-second responses.

Fine-tuning can improve latency but narrow the model too much.

Root cause: The fine-tuned Claude 3 Haiku experiment performed well but became specialized to static use cases.

Resolution: Treat fine-tuning as one option in the model lifecycle and keep evaluating broader models and approaches.

Evaluation

Evaluation combines human tax-expert review, manual golden data, LLM-as-judge automation, production sampling, and monitoring. Major tax-year or prompt changes return to manual evaluation, while smaller prompt iterations use automated evaluation based on expert guidance.

Metrics: accuracy · relevancy · coherence · tax-accurate output · manual expert review · LLM-as-judge automated evaluation · production sample monitoring

Outcomes

• Intuit deployed TurboTax LLM explanations through Intuit Assist and GenOS
• Claude is used for the production static explanation use case
• RAG and Graph RAG support personalized tax-answer quality
• fine-tuned Claude 3 Haiku performed well but was judged too specialized for the specific use case
• clear evaluation enabled migration from Claude Instant to Claude Haiku for tax year 2024

The assistant operates in a TurboTax environment that processed 44 million tax returns for tax year 2023 and experiences peak load around the April 15 deadline.