Post-Training a Frontier Legal AI Agent Through Full-Stack Optimization

harvey · Legal · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Legal AI agents and post-training infrastructure

Harvey and Applied Compute needed to train an open-weight legal agent that could satisfy strict legal benchmark criteria where partial success is not enough, while ensuring the grader signal could be trusted, long-document tasks could fit through an agent harness, and tool use became efficient and grounded rather than brute-force.

Constraints:

• Legal tasks fail if a model misses a clause, overlooks a document, misstates a jurisdictional detail, or fails one step of a multi-step instruction.
• LAB tasks require all required criteria to pass, not just a fluent or mostly correct answer.
• The benchmark contains more than 1,250 tasks, 24 legal practice areas, and more than 75,000 binary criteria.
• A powerful model can exploit flaws in grader criteria or the evaluation environment.
• Large source documents caused around 10% of base GLM-5.1 rollouts to reach maximum context length.
• The 90th-percentile LAB datapoint contains nearly 100,000 source-document tokens, with maximums above 200,000.
• Scaling reinforcement learning required a cheaper but aligned grader setup.

Architecture

Applied Compute and Harvey optimized the whole legal-agent training stack. They established trusted grader signals by comparing frontier graders and cheaper candidate graders against a gold-trace consensus; reduced grader cost by batching criteria through GPT-5 Mini; improved Harvey's LAB harness with tool-output limits, broken-tool-call repairs, tool-use advice, output reminders, and compaction; trained GLM-5.1 with full-parameter fully-asynchronous reinforcement learning on AC2; and analyzed the resulting model for learned behaviors such as artifact completeness, specificity/exactness, grounding, and more efficient tool use.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Legal_Agent_Benchmark["Legal Agent Benchmark"] --> Agent_harness["Agent harness"]
  Legal_Agent_Benchmark --> Grader_evaluation_system["Grader evaluation system"]
  AC2_platform["AC2 platform"] --> Agent_harness
  AC2_platform --> Grader_evaluation_system
  AC2_platform --> Full_parameter_RL_trainer["Full-parameter RL trainer"]
  Agent_harness --> GLM_5_1_agent["GLM-5.1 agent"]
  Agent_harness --> Compaction_module["Compaction module"]
  Agent_harness --> Legal_source_documents["Legal source documents"]
  Compaction_module --> GLM_5_1_agent
  Grader_evaluation_system --> GPT_5_Mini_grader["GPT-5 Mini grader"]
  Grader_evaluation_system --> Frontier_grader_consensus["Frontier grader consensus"]
  Full_parameter_RL_trainer --> GLM_5_1_agent
  Full_parameter_RL_trainer --> Behavior_analysis["Behavior analysis"]

Components

Data flow

LAB tasks provide source documents, instructions, and task-specific rubrics. Candidate agent rollouts run in a sandboxed harness with read, grep, bash, docx generation, and verification tools. When context grows too large, the harness summarizes the episode transcript and starts a fresh conversation from that summary. Outputs and traces are graded by an aligned LLM-as-judge setup using batched criteria calls. AC2 runs evaluations, consistency checks, RL training, and behavioral analysis, then compares baseline and trained checkpoints on held-out tasks.

Why this architecture

The team optimized grader, harness, and model together because LAB's strict legal tasks make weak evaluation signals and poor tool scaffolding dangerous. A trusted and affordable grader made RL scalable; harness improvements made tool-using rollouts feasible; compaction let the agent work beyond native context limits; and full-parameter RL taught GLM-5.1 to use tools and documents better rather than only relying on prompt scaffolding.

Models used: GLM-5.1 · AC GLM-5.1 · GPT-5.5 xhigh · Opus 4.8 Max · Claude Opus 4.7 Max · GPT-5 Mini · Claude Sonnet 4.6 · GPT-5 Nano · Claude Haiku 4.5 · Qwen 3.6 35B A3B · Kimi K2.6

Scale

• LAB contains more than 1,250 tasks.
  “LAB contains more than 1,250 tasks across 24 legal practice areas.”article, stated
• LAB spans 24 legal practice areas.
  “LAB contains more than 1,250 tasks across 24 legal practice areas.”article, stated
• The benchmark contains more than 75,000 binary criteria.
  “the benchmark contains more than 75,000 binary criteria.”article, stated
• The grader analysis used a 50-point subset with over 2,500 rubric criteria.
  “ran a 50-point subset through the original LAB harness, producing traces covered by over 2,500 rubric criteria.”article, stated
• GPT-5.5 xhigh and Claude Opus 4.7 Max agreed on over 95% of criteria across runs.
  “GPT-5.5 xhigh and Claude Opus 4.7 Max agreed on over 95% of criteria across all runs.”article, stated
• The top four graders had a self-consistency floor of 98.9%.
  “We found a floor of 98.9% per-criteria self-consistency”article, stated
• Batching GPT-5 Mini grader calls saved 40x to 100x cost relative to the original setup.
  “we observed between 40x (at 16-per-group) and 100x (at all-criteria-in-one-group) cost savings relative to the original setup.”article, stated
• About 10% of base GLM-5.1 rollouts reached maximum context length.
  “around 10% with the base GLM-5.1 model) reach the max context length limit”article, stated
• The 90th-percentile datapoint contains nearly 100,000 source-document tokens.
  “the 90th-percentile datapoint contains nearly 100,000 tokens in source documents”article, stated
• Some source documents exceed 200,000 tokens.
  “with the maximum rising to over 200,000 tokens of source documents.”article, stated
• The final compaction harness triggers at 131,072 context tokens and allows up to four compactions.
  “we use a compaction trigger of context_tokens >= 131072 and up to 4 max_compactions.”article, stated
• The trained model improved rubric pass rate from 0.853 to 0.913.
  “lifts rubric pass rate from 0.853 to 0.913”article, stated
• The trained model improved all-pass rate from 0.059 to 0.126.
  “all-pass rate from 0.059 to 0.126”article, stated

Alternatives considered (roads not taken)

Trust the original benchmark grader setup without analysis

Rejected because: The team believed a powerful model would exploit flawed grader signals, so they established frontier-model consensus and candidate-grader consistency first.

Grade one criterion per call with Claude Sonnet 4.6

Rejected because: GPT-5 Mini with batched criteria kept high alignment while cutting grader costs by 40x to 100x.

Use the base LAB harness unchanged

Rejected because: The team found scalable improvements through token-output restrictions, tool-call repair, tool-use advice, output reminders, and compaction.

Rely on native context length

Rejected because: Large legal files caused rollouts to hit context limits, so compaction extended effective working context while forcing better context use.

Train with parameter-efficient or scaffolding-only changes

Rejected because: GLM-5.1 did not meaningfully improve from harness updates alone and needed full-parameter RL to improve tool use and failure modes.

Patterns

fine-tuning-adaptation (model-adaptation)

Applied Compute post-trained GLM-5.1 with full-parameter reinforcement learning on legal-agent tasks to improve tool use, grounding, exactness, and artifact completion.

eval-driven-development (evaluation)

The whole training process is organized around held-out LAB evaluation, grader alignment, self-consistency, cross-consistency, rubric pass rate, all-pass rate, and behavioral analysis.

cost-aware-routing (cost)

The team chose a cheaper aligned grader and batched criteria per grader call to make high-compute RL grading affordable.

tool-use-orchestration (agent-architecture)

The legal agent uses read, grep, bash, document-generation, and verification tools, and training shifts it toward targeted tool use.

context-compaction (context)

The harness summarizes the episode transcript at a token threshold, starts a fresh conversation from the summary, and permits up to four compactions.

observability-tracing (operations)

AC2 analyzes traces, training/eval scores, tool-use patterns, domain shifts, and behavior-level failure-to-pass flips to interpret what the model learned.

model-distillation (model-adaptation)

The conclusion identifies relevance-masked self-distillation as a future complementary technique to further strengthen grounding.

Tradeoffs

Optimize the full stack rather than only the model.

Chose grader analysis, harness engineering, compaction, and full-parameter RL over model training alone.

Rationale: The model could only improve safely and affordably if the grader signal, tool harness, and long-context management were also reliable.

Use a cheaper batched grader setup.

Chose GPT-5 Mini with batched criteria-per-call over the original one-criterion-per-call Claude Sonnet 4.6 setup.

Rationale: The batched setup preserved alignment above 97% while making RL grading and sampling economically scalable.

Use compaction to extend legal-agent work.

Chose automatic transcript summarization and fresh conversations over terminating or truncating long-context rollouts at native context length.

Rationale: LAB tasks include very large documents, and compaction lets the agent continue while learning to avoid context rot.

Use full-parameter RL for GLM-5.1.

Chose full-parameter fully-asynchronous reinforcement learning over harness-only improvements or lighter adaptation.

Rationale: GLM-5.1's baseline performance was strong, but harness changes alone did not improve it; RL taught it better tool usage and document reasoning.

Gotchas (what bit them)

A powerful model can optimize against grader flaws instead of legal quality.

Root cause: LAB uses LLM-as-a-Judge signals, and RL will exploit loopholes in criteria or environment if the signal is weak.

Resolution: Build a gold-trace ground truth from frontier-grader agreement and validate cheaper graders against self- and cross-consistency.

Naive grader calls make high-compute RL too expensive.

Root cause: The original setup evaluated one criterion per call with a larger grader model.

Resolution: Use GPT-5 Mini and batch multiple criteria per call after confirming high alignment with frontier ground truth.

Legal-agent rollouts can overflow context through broad file reads.

Root cause: Large source files and tool outputs dump hundreds of thousands of tokens into the context window.

Resolution: Limit tool-output tokens, encourage targeted reads, and use compaction to summarize long transcripts before continuing.

Harness improvements do not guarantee capability gains for every base model.

Root cause: GLM-5.1 did not materially improve from the harness update alone because it needed to learn better tool use and failure-case handling.

Resolution: Train the model with full-parameter RL after confirming harness parity and improvements.

Legal answers fail when they are broad, rounded, or insufficiently grounded.

Root cause: The base model rounded numbers, hallucinated source items, invented findings, and made poorly supported claims.

Resolution: RL improved artifact completeness, specificity/exactness, and grounding, flipping hundreds of criteria from fail to pass.

Evaluation

Evaluation uses LAB's held-out test split, task-specific binary rubrics, rubric pass rate, all-pass rate, grader alignment against frontier consensus, self-consistency across reruns, cross-consistency correlations, domain-level score shifts, and behavior-level failure-to-pass analysis.

Metrics: Rubric pass rate · All-pass rate · Per-criteria grader alignment · Per-criteria self-consistency · Cross-consistency Pearson correlation · Cost per trace · Domain-level score shift · Tool calls · Read calls · Bash calls · Tool payload tokens · Criteria flipped by artifact completeness · Criteria flipped by specificity/exactness · Criteria flipped by grounding

Outcomes

• The trained GLM-5.1 model achieved a rubric pass rate of 0.913, outperforming available models by rubric pass rate according to the article.
• All-pass rate improved from 0.059 to 0.126.
• GPT-5 Mini with batched criteria calls enabled 40x to 100x grader cost savings.
• Training reduced sample task tool calls from 104 to 42 and payload tokens from 461k to 250k.
• Artifact completeness flipped 185 relevant criteria, specificity/exactness flipped 243, and grounding flipped 70.
• The trained model learned to make more targeted read and bash calls, generate complete artifacts, and make more specific grounded legal claims.

The work targets a legal benchmark with more than 1,250 tasks, 24 practice areas, and over 75,000 binary criteria. The final system uses compaction at 131,072 context tokens with up to four compactions to handle source documents that can exceed 200,000 tokens.