Evaluating Long Context Performance in Legal AI Applications

thomson_reuters · Legal · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: long-context legal AI evaluation

Thomson Reuters needed to evaluate whether long-context LLMs can reliably handle lengthy legal documents and multi-document tasks in CoCounsel, despite advertised context windows overstating effective performance on complex legal reasoning.

Constraints:

Legal documents can exceed hundreds of pages.
Some legal tasks require simultaneous analysis of multiple documents.
Advertised context windows up to 1M tokens do not guarantee accurate use of all context.
Complex tasks require tracking many data elements and cross-document relationships.
Legal reliability requires attorney SME review before production deployment.

Architecture

Thomson Reuters uses a multi-stage LLM evaluation process for CoCounsel. Initial automated benchmarks cover legal reasoning, contract understanding, hallucinations, instruction following, and long context. Top models then move into skill-specific tests using attorney-authored use cases, ideal answers, and LLM-as-a-judge prompts calibrated against SME scores. Final deployment requires manual attorney review. CoCounsel 2.0 generally uses long-context LLMs for document-based skills, while reserving RAG for repository-search skills.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  cocounsel-document-tasks["CoCounsel document tasks"] --> initial-automated-benchmarks["Initial automated benchmarks"]
  initial-automated-benchmarks --> skill-specific-test-suites["Skill-specific test suites"]
  skill-specific-test-suites --> llm-as-a-judge-grader["LLM-as-a-judge grader"]
  llm-as-a-judge-grader --> attorney-sme-review["Attorney SME review"]
  attorney-sme-review --> production-model-flow["Production model flow"]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`CoCounsel document tasks`	Legal workflows such as contract, transcript, and document analysis.	`Initial automated benchmarks`
`Initial automated benchmarks`	Over 20,000 test samples across legal reasoning, contracts, hallucinations, instructions, and long context.	`Skill-specific test suites`
`Skill-specific test suites`	Attorney-authored use cases, source documents, and ideal minimum viable answers for each skill.	`LLM-as-a-judge grader`
`LLM-as-a-judge grader`	Scores skill outputs against SME-authored ideal answers using calibrated grading prompts.	`Attorney SME review`
`Attorney SME review`	Manually reviews grader alignment, nuanced errors, and final model-flow readiness.	`Production model flow`
`Production model flow`	Chosen long-context or RAG-backed CoCounsel skill flow deployed for legal users.	—

Data flow

Candidate LLMs are screened on broad automated benchmark samples, then evaluated on skill-specific legal test suites with source documents and attorney-authored ideal answers. LLM judges score model outputs against those answers, SME reviewers calibrate grader prompts and final decisions, and the chosen model flow is deployed only after manual legal review.

“before deploying an LLM into production, we conduct multiple stages of testing, each more rigorous than the last, to ensure peak performance.”article, stated

“Once a skill flow is fully developed, it undergoes evaluation using LLM-as-a-judge against attorney-authored criteria.”article, stated

Why this architecture

Thomson Reuters uses long-context models for most document-based skills because internal tests found full-document context generally beat RAG for legal document tasks requiring complex comparisons. But the company keeps a multi-LLM strategy and staged attorney-calibrated evaluations because effective context windows shrink on difficult tasks and legal errors have high professional stakes.

“we found that inputting the full document text into the LLM’s input window (and chunking extremely long documents when necessary) generally outperformed RAG for most of our document-based skills.”article, stated

“Our legal, tax, and compliance workflows are complex, unforgiving, and grounded in real-world stakes”article, stated

Models used: CoCounsel · Long-context LLMs · GPT-4 · GPT-4.1 · o1-mini custom model · LLM-as-a-judge · RAG

Scale

GPT-4's initial 8K context was about 6,000 words or 20 pages.
“GPT-4 was first released in 2023, it featured a context window of 8K tokens, equivalent to approximately 6,000 words or 20 pages of text.”article, stated
Current major LLMs advertise context windows from 128K to over 1M tokens.
“Today, most major LLMs have context windows ranging from 128K to over 1M tokens.”article, stated
Initial benchmarks use over 20,000 test samples.
“We use over 20,000 test samples from open and private benchmarks covering legal reasoning, contract understanding, hallucinations, instruction following, and long context capability.”article, stated
Specialized long-context tests cover inputs from 100K to 1M tokens.
“we have constructed specialized long context test sets where all test samples use one or more source documents totaling 100K–1M tokens in length.”article, stated

Alternatives considered (roads not taken)

Use RAG for most document-based legal tasks.

Rejected because: Complex legal queries can require comparing many statements across a full document, where semantic retrieval may not retrieve all needed evidence.

“some complex queries require a more sophisticated discovery process and more context from the underlying documents.”article, stated

Trust advertised context-window size.

Rejected because: Effective context can be much smaller than available context, especially on complex tasks.

“the effective context windows of LLMs, where they perform accurately and reliably, are often much smaller than their available context window.”article, stated

Rely only on automated graders.

Rejected because: Attorney SMEs catch nuanced legal details missed by automated grading and provide the final production check.

“Our SMEs can capture nuanced details missed by automated graders and provide feedback to our engineers for improvement.”article, stated

Patterns

`eval-driven-development` (legal AI deployment)

Thomson Reuters uses staged benchmarks, skill-specific tests, judge calibration, and SME review before deploying LLM flows.

“conduct multiple stages of testing, each more rigorous than the last, to ensure peak performance.”article, stated

`llm-as-judge` (legal skill scoring)

LLM judges score CoCounsel skill outputs against attorney-authored ideal answers and criteria.

“SMEs and engineers collaborate to create grading prompts so that an LLM judge can score skill outputs against the ideal answers written by our SMEs.”article, stated

`human-in-the-loop-gate` (legal review)

Attorney SMEs manually review new LLM flows before production deployment.

“All new LLMs undergo rigorous manual review by our attorney SMEs before deployment.”article, stated

`model-routing` (multi-LLM strategy)

The organization tracks major open and closed model releases and uses a multi-LLM strategy for capabilities like long context.

“We track and evaluate all major LLM releases, both open and closed source, to ensure we are using the latest and most advanced models”article, stated

Tradeoffs

Prefer long context for document-based skills.

Chose Full document text in the LLM input window, with chunking for extremely long documents over RAG for most document-based legal skills.

Rationale: Internal testing found long context generally performed better for document-based skills requiring broad context.

“Consequently, in CoCounsel 2.0 we leverage long context LLMs to the greatest extent possible to ensure all relevant context is passed to the LLM.”article, stated

Use automated benchmarks first.

Chose Large gradable benchmark screening over Sending every candidate model directly to attorney review.

Rationale: Over 20,000 automated samples provide fast initial filtering before more expensive skill-specific and manual legal evaluation.

“These tests have easily gradable answers (e.g., multiple-choice questions), allowing for full automation and easy evaluation of new LLM releases.”article, stated

Keep final SME review.

Chose Attorney manual review before deployment over Automated judge-only deployment.

Rationale: Legal workflows are high-stakes and require nuanced professional validation.

“provide the final check to verify that the new LLM flow performs better than the previously deployed solution and meets the exacting standards for reliability and accuracy in legal use.”article, stated

Gotchas (what bit them)

Long advertised context does not mean reliable long-context reasoning.

Root cause: Accuracy degrades as input length and reasoning complexity increase.

Resolution: Build specialized long-context tests at realistic legal lengths and chunk extremely long documents when needed.

“the ability to fit 1M tokens into an input window does not guarantee effective performance with that much text.”article, stated

RAG can miss evidence for comparison-heavy legal questions.

Root cause: Semantic retrieval may return passages about a topic but not every statement needed for contradiction or multi-document analysis.

Resolution: Use full-document long context where appropriate and reserve RAG for repository search.

“the query “Did the defendant contradict himself in his testimony?” requires comparing each statement in the testimony against all others”article, stated

LLM judge scores need calibration.

Root cause: Legal answer quality depends on attorney-authored criteria and nuanced professional judgment.

Resolution: SMEs and engineers iteratively adjust grading prompts and ideal answers until judge scores align with SME scores.

“LLM-as-a-judge scores are manually reviewed, grading prompts are adjusted, and ideal answers are refined until the LLM-as-a-judge scores align with our SME scores.”article, stated

Evaluation

The evaluation stack uses automated broad benchmarks, LOFT and NovelQA long-context tests, skill-specific attorney-authored test suites, LLM-as-a-judge grading calibrated to SME scores, and final manual attorney review.

Metrics: Legal reasoning · Contract understanding · Hallucination behavior · Instruction following · Long-context capability · Multihop reasoning · Multitarget reasoning · SME alignment

“Both tests accommodate up to 1M input tokens and measure key long context capabilities critical to our skills, such as multihop reasoning”article, stated

“and an ideal minimum viable answer capturing the key data elements necessary for the answer to be useful in a legal context.”article, stated

Outcomes

Thomson Reuters found long-context LLMs generally outperformed RAG for most document-based CoCounsel skills.
The team identified that effective context windows are often smaller than advertised windows.
The evaluation process supports a multi-LLM product strategy and early testing of new long-context models.

The case gives evaluation scale rather than deployment usage: over 20,000 initial benchmark samples, long-context tests up to 1M tokens, and specialized sets of 100K-1M token inputs.

“When you look at the advertised context window for leading models today, don’t be fooled into thinking this is a solved problem.”article, stated

“And when the next generation of long-context models hits the market, we are routinely early testers. A multi-model strategy only works if you know which model to trust—and when.”article, stated