Building an AI Legal Assistant: From Early Testing to Production Deployment

casetext · Legal · 2023 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Legal AI assistant

Casetext needed to turn GPT-4's breakthrough legal reasoning into a production legal assistant accurate enough for mission-critical legal research and document review.

Constraints:

• Lawyers would ignore incremental improvements unless the product was dramatically better than existing tools.
• Legal work requires citations, accuracy, and no hallucinated assumptions.
• Complex legal workflows had to be decomposed into reliable steps rather than one broad prompt.
• The company had to pivot a 120-person organization very quickly after early GPT-4 access.

Architecture

Casetext built Co-Counsel as a set of legal 'skills' rather than one monolithic chat prompt. For a legal research task, the system decomposes the lawyer's request into search queries, executes them against legal databases, reads and summarizes returned cases, extracts cited insights, and assembles a memo. Each prompt in the chain is developed test-first with gold-standard expected outputs and thousands of regression cases.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  lawyer[Lawyer] --> co-counsel[Co-Counsel]
  co-counsel --> skill-chain[Skill Chain]
  skill-chain --> prompt-steps[Prompt Steps]
  skill-chain --> legal-databases[Legal Databases]
  prompt-steps --> test-harness[Test Harness]
  prompt-engineers[Prompt Engineers] --> test-harness[Test Harness]
  prompt-engineers --> prompt-steps[Prompt Steps]

Components

Data flow

A lawyer gives Co-Counsel a task such as reviewing documents or researching a legal question; the system breaks it into legal substeps, runs database searches, reads case/statute/regulation results, notes and summarizes findings, composes a memo with citations, and each prompt step is guarded by a battery of tests.

Why this architecture

Legal work demanded reliable intermediate reasoning, citations, and regression safety. Casetext therefore modeled how a strong attorney would work, decomposed each job into promptable steps, and made prompt engineering test-driven so fixing one failure would not silently break another legal subtask.

Models used: GPT-4 · GPT-3.5

Scale

• Casetext pivoted a 120-person company within roughly 48 hours.
  “the time so you took like 120 people and completely changed what they were all working on yes yes yes in 48 hours yes and for the people watching uh casex originally I mean had always been in the legal space you're a lawyer and you built something for yourself and you”transcript, stated
• The company had grown 70-80% year over year and reached $15-20M ARR before the pivot.
  “product and we're making you know we're growing 70 80% year-over-year we're between $15 and $20 million in ARR things weren't like terrible right um that's great yeah we're great yeah we but like so they were like what why are we even the board you know some of the”transcript, stated
• GPT-4 scored better than 90% of test takers in a new bar-passage study, versus GPT-3.5 at the 10th percentile.
  “and it showed that gbd 3.5 got a 10 10th percentile on the bar passage right so like it did better than some people actually but the 10% of them yeah probably the ones who just filling out randomly basically um when we got early”transcript, stated

  “totally new test to it and the test we ran it did better than 90% of the test takers right so it's like a big difference and and also we started runting some tests like okay here's like four or five cases to read um using those cases write a memo”transcript, stated
• Co-Counsel launched before conversations that led to a $650M Thomson Reuters acquisition.
  “matter of 2 months after the release of GPT 4 that valuation went to a liquid exit to Thompson Reuters for $650 million so you have a lot of lessons about how to create real value from really like large language models I think you were of um you know our friends in YC one of the first people to”transcript, stated
• Research skills could involve hundreds or thousands of cases.
  “memo after researching you know hundreds or thousands of cases answering the lawyer's initial research question and and so in that sense it was this like really powerful extension of the workforce of these law firms that was the concept from the beginning and we”transcript, stated

Alternatives considered (roads not taken)

Continue incremental legal NLP and recommendation features.

Rejected because: They generated revenue but were not transformative enough to overcome lawyer inertia.

Ask one large prompt to solve full legal research tasks.

Rejected because: The team modeled attorney work as a chain of search, reading, note-taking, citation, and memo-writing steps.

Vibes-only prompt engineering.

Rejected because: Legal reliability required tests for each prompt because instructions can fix one issue and break another.

Patterns

prompt-chaining (workflow)

Legal skills are decomposed into a dozen or two dozen prompt steps that mirror expert attorney workflow.

eval-driven-development (quality)

Prompt engineers wrote tests before prompt changes and used batteries of cases to prevent regressions.

structured-output-contract (legal-workflow)

Each prompt step had a clear expected output such as search queries or cited memo sections.

Tradeoffs

Company focus

Chose Pivot all 120 people to GPT-4 legal assistant work over Continue the existing NLP business trajectory.

Rationale: GPT-4 produced a step-function capability increase in legal tasks.

Legal workflow design

Chose Skill chains with tested intermediate prompts over Single end-to-end prompt.

Rationale: Decomposition followed how expert attorneys actually perform legal research.

Prompt engineering method

Chose Test-driven prompt development over Manual prompt tweaking by feel.

Rationale: High-stakes legal work needed regression confidence at every prompt step.

Gotchas (what bit them)

Legal GPT work cannot tolerate plausible hallucination.

Root cause: Legal users need accurate citations and assumptions grounded in the provided context.

Resolution: Prompt toward actual context citations and validate steps with tests.

Fixing one prompt behavior can break another.

Root cause: LLMs may go in unexpected directions after new instructions are added.

Resolution: Run prompt changes against large regression batteries for each skill step.

Transformative AI required founder-led organizational conviction.

Root cause: The company already had meaningful ARR, so not everyone immediately agreed to pivot.

Resolution: The founder built the first version himself and led by example.

Evaluation

Casetext evaluates legal assistant behavior through prompt-level gold-standard expected outputs, dozens to thousands of tests per prompt, bar-exam style studies, and regression batteries for each legal skill step.

Metrics: Bar-passage percentile · Prompt test pass rate · Gold-standard search-query match · Regression failures by prompt step · Legal task completion time

Outcomes

• Co-Counsel performed legal document review and research tasks that previously took lawyers days.
• Customers tested early GPT-4-backed versions under NDA before public launch.
• Casetext's valuation moved from around $100M to a $650M Thomson Reuters exit after the GPT-4 pivot.
• The team established test-driven prompt engineering as the core production method.

The source cites 120 employees, 48-hour company pivot, 70-80% YoY growth, $15-20M ARR, $100M to $650M valuation/exit change, 10th vs 90th percentile bar results, hundreds/thousands of cases, and prompt tests scaling from dozens to thousands.