Graph RAG and Multi-Agent Systems for Legal Case Discovery and Document Analysis

whyhow · Legal · year n/a · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: legal case discovery and mass-tort research

WhyHow needed to find potential class-action and mass-tort cases earlier than lawyers could manually, while transforming web complaints and discovery documents into accurate, lawyer-specific, legally usable reports and queryable evidence graphs.

Constraints:

• legal work requires accuracy, correct language, and domain-specific creative reasoning
• probabilistic LLM agents alone are not reliable enough for high-stakes legal workflows
• multi-step agent chains compound error rates
• lawyers have different case preferences, schemas, jurisdictions, and report formats
• discovery can involve document mountains such as 500 GB of emails
• web complaints appear across many sources including government websites, carcomplaints.com, Reddit, Twitter, forums, and other communities
• domain experts must be able to inspect and decide from structured, visual, and natural-language evidence

Architecture

WhyHow combines web scraping, ML filtering, LLM-enabled multi-agent workflows, lawyer-specific schemas, and graph-backed state. The system scrapes broad web sources, filters and qualifies signals against each lawyer's schema, builds and updates graph structures for products, harms, ingredients, concentrations, people, complaints, and lawsuits, then uses graph RAG and report-generation agents to produce lawyer-specific actionable reports. Discovery workflows similarly extract and structure information from large document sets so experts can query, visualize, and decide from relevant subgraphs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  web_and_discovery_sources["Web and discovery sources"] --> scraping_and_ingestion_layer["Scraping and ingestion layer"]
  scraping_and_ingestion_layer --> ml_filtering_layer["ML filtering layer"]
  scraping_and_ingestion_layer --> graph_construction_pipeline["Graph construction pipeline"]
  ml_filtering_layer --> multi_agent_workflow["Multi-agent workflow"]
  graph_construction_pipeline --> lawyer_specific_graph_state["Lawyer-specific graph state"]
  lawyer_specific_graph_state --> multi_agent_workflow
  lawyer_specific_graph_state --> graph_rag_query_layer["Graph RAG query layer"]
  multi_agent_workflow --> report_generator["Report generator"]
  graph_rag_query_layer --> report_generator
  multi_agent_workflow --> guardrails_and_human_review["Guardrails and human review"]
  guardrails_and_human_review --> lawyer_or_domain_expert["Lawyer or domain expert"]
  graph_rag_query_layer --> lawyer_or_domain_expert
  report_generator --> lawyer_or_domain_expert

Components

Data flow

The system scrapes the web or ingests discovery documents, extracts and structures information into graphs, filters signals through lawyer-specific schemas, and manages state by expanding, pruning, and querying graph structures. Multi-agent workflows perform stepwise qualification and report generation, with guardrails and human-in-the-loop review where legal accuracy matters. Lawyers receive personalized reports or natural-language graph queries that expose only the signals and subgraphs they care about.

Why this architecture

Legal case discovery needs both precise structure and expert-specific flexibility. Graphs provide explicit relationships, multihop analytics, visual inspection, and extensible schemas; ML filters reduce noisy web scale; LLMs let formerly incompatible workflow steps pipe together; and multi-agent decomposition makes each step testable while guardrails and humans compensate for compounded agent uncertainty.

Models used: LLMs · traditional ML systems · multi-agent systems

Scale

• A legal discovery corpus can include 500 GB of emails.
  “Here is 500 GB of emails that don't matter. Go nuts, right? Figure out exactly what happened”transcript, stated
• The speaker estimates that five sequential agents at 95% accuracy produce about 77% expected end-to-end accuracy.
  “five 95% accurate agents chained together sequentially. That's 77% expected accuracy.”transcript, stated
• The system can find some case signals within about 15 minutes, though confidence may take about a month.
  “for us, we can find it uh within about 15 minutes and then generally it takes probably a month for you to be confident”transcript, stated
• Traditional large-law-firm case discovery may occur eight or nine months after complaints start.
  “for a large law firm, maybe eight or nine months post people starting to complain, they can take that lawsuit on”transcript, stated
• The graph may contain millions of nodes while only a few entities are legally relevant.
  “There is only going to be a few people in that graph of potentially millions of nodes that are a problem”transcript, stated

Alternatives considered (roads not taken)

Use probabilistic LLM agents in isolation

Rejected because: Legal workflows require exact language and correctness, and isolated probabilistic agents are not trustworthy enough for that standard.

Rely on a chat RAG interface as the primary legal product

Rejected because: Lawyers appreciate answers, but they also need familiar report formats generated from retrieved subgraphs.

Search web complaints with generic best-case ranking

Rejected because: There is no universal perfect lawsuit; each lawyer or firm needs personalized schemas and filters.

Manually review discovery documents end to end

Rejected because: Discovery corpora can be physically impossible to read within lawsuit timelines.

Patterns

rag-retrieval (graph-backed legal retrieval)

WhyHow retrieves lawyer-specific subgraphs from structured legal data and uses those subgraphs to answer questions and generate actionable reports.

multi-agent-orchestration (legal workflow decomposition)

The system decomposes legal work into multiple agent steps with different state, frequency, and requirements rather than using one large prompt.

agent-memory (graph state management)

WhyHow stores episodic and workflow state in graphs, then captures, expands, prunes, structures, and queries that state over time.

human-in-the-loop-gate (legal expert decision)

The system presents graph-derived evidence and reports to lawyers or domain experts so humans make legal decisions from structured signals.

eval-driven-development (stepwise agent quality)

WhyHow designs multi-agent steps so they can be input/output tested and so failures can be pinpointed instead of hidden inside an end-to-end chain.

structured-output-contract (lawyer-specific reports)

Reports are generated from consistent schemas and personalized graph state, preserving a stable structure for each lawyer while adapting content to their case preferences.

guardrails-io-filtering (agent reliability)

The team spends significant effort guardrailing agents because a badly prompted agent or chained agent workflow can be wrong in legally unacceptable ways.

Tradeoffs

Use graphs as controlled state.

Chose Graph-backed episodic and workflow state over Immediate prompt context or unstructured memory alone.

Rationale: Graphs make state extensible, prunable, structured, queryable, and visually inspectable for high-stakes legal workflows.

Combine ML filters with LLM pipe-fitting.

Chose ML-filtered system where LLMs connect probabilistic workflow steps over An LLM-filtered system alone.

Rationale: Traditional ML handles filtering where it is strong, while LLMs make previously incompatible workflow steps composable end to end.

Generate reports rather than only chat answers.

Chose Graph-RAG report generation in lawyer-specific formats over Chat RAG as the main interface.

Rationale: Lawyers need familiar actionable form factors, not just conversational answers.

Personalize schemas per lawyer.

Chose Different multi-agent system and graph backing each lawyer's workflow over A single shared legal-discovery schema for everyone.

Rationale: Each lawyer cares about different filters, report forms, and case criteria, so personalization is core to usefulness.

Use humans and guardrails to raise usable accuracy.

Chose Guardrails plus humans in the loop over Relying on agent accuracy alone.

Rationale: The speaker does not trust individual agents enough for legal work, and sequential chains compound error risk.

Gotchas (what bit them)

Chaining good-but-imperfect agents quickly degrades end-to-end reliability.

Root cause: A five-step workflow with individually 95% accurate agents has only about 77% expected end-to-end accuracy.

Resolution: Break workflows into testable steps, add guardrails, graph state, and human checks rather than trusting a long opaque chain.

Legal users do not tolerate casual probabilistic errors.

Root cause: Lawyers require correctness, proper language, and traceable detail before creative argumentation can be useful.

Resolution: Use explicit schemas, graph structure, tests, guardrails, and expert review around LLM/agent output.

Generic graph visuals can create false shared understanding.

Root cause: Different people can look at the same graph and think they understand it while interpreting it differently.

Resolution: Define the graph schema explicitly and keep lawyer-specific structure consistent.

Discovery corpora contain far more irrelevant material than useful evidence.

Root cause: Opposing parties may provide huge volumes of documents, and lawsuit timelines make manual full review impossible.

Resolution: Extract and structure the corpus into graphs so irrelevant information can be dismissed and useful subgraphs surfaced visually and through language.

A universal ranking of case leads does not fit legal practice.

Root cause: Whether a case is valuable depends on the lawyer, partner, firm, jurisdiction, money amount, and preferred theory.

Resolution: Maintain lawyer-specific schemas and personalized report generation.

Evaluation

WhyHow evaluates through stepwise input/output testing of multi-agent workflows, explicit tracking of where a workflow goes right or wrong, agent-chain accuracy reasoning, signal velocity/density over time, lawyer-specific feedback, and confidence growth after initial lead detection.

Metrics: single-agent accuracy example: 95% · five chained 95% agents produce 77% expected accuracy · discovery example: 500 GB of emails · case signal can be found within about 15 minutes · confidence in a lead can take about a month · large law firm timing can be eight or nine months after complaints start · complaint density uses count multiplied by complaint velocity over months

Outcomes

• WhyHow reports production agents that lawyers can query in natural language to understand lawsuits and represented individuals.
• The system turns broad web scraping into lawyer-specific qualified signals.
• Graph-backed report generation produces personalized reports that lawyers can action.
• Discovery document mountains can be transformed into structured graph views that highlight relevant people, products, ingredients, concentrations, timing, and harms.
• For some case research signals, WhyHow says it can identify potential leads within about 15 minutes and build confidence over about a month.
• The architecture enables end-to-end value by combining ML filters, LLM-powered composition, graph state, and multi-agent workflows.

The source describes web-scale scraping, discovery corpora such as 500 GB of emails, potentially millions of graph nodes, and early lead identification measured in minutes to months rather than law-firm timelines of eight or nine months after complaints begin.