Rapid Post-Training of Open-Weight Models for Legal AI Applications

trajectory · Legal · 2026 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Legal AI model post-training and evaluation

Trajectory needed to adapt newly released open-weight models to complex legal work quickly enough to keep pace with foundation-model releases while matching frontier closed-model quality at lower serving cost.

Constraints:

• Legal work requires high completeness across rubric criteria, not just partial task success.
• New open-weight models arrive frequently, so adoption cannot require re-engineering the training pipeline each time.
• The system needed held-out legal-task evaluation rather than only training-set performance.
• Cost mattered because reasoning-heavy, long-horizon legal tasks can make frontier closed models expensive to run.

Architecture

Trajectory used a model-agnostic learning layer and the Harvey Legal Agent Bench (LAB) to post-train NVIDIA Nemotron 3 Ultra in under 24 hours. The same harness, data pipeline, and recipe previously used for Nemotron 3 Super were pointed at the new model weights, producing a post-trained open-weight model evaluated on held-out LAB tasks. The evaluation measured all-pass rate, individual rubric criteria passed, per-task score distributions, capability gains, and quality-vs-cost placement against closed frontier models.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["Nemotron 3 Ultra base model"] --> B["Trajectory learning layer"]
  B --> C["Training harness"]
  B --> D["Legal data and recipe"]
  C --> E["Post-trained Nemotron 3 Ultra"]
  D --> C
  E --> F["Harvey LAB evaluation"]
  F --> G["Quality cost analysis"]
  F --> H["Capability analysis"]

Components

Data flow

A new open-weight base model arrives. Trajectory points its existing post-training harness, legal data pipeline, and training recipe at the new weights, runs domain-specific post-training, and evaluates the resulting model on held-out Harvey LAB tasks. Evaluation aggregates all-pass rates, rubric coverage, score distributions, capability gains, and cost/quality positioning to decide whether the model is production-useful for legal work.

Why this architecture

The key architecture choice is a decoupled, model-agnostic learning layer: it lets Trajectory reuse the same harness, data, and recipe whenever a stronger open model appears. That turns foundation-model churn into a repeatable post-training operation rather than a custom engineering project.

Models used: NVIDIA Nemotron 3 Ultra · NVIDIA Nemotron 3 Super · Sonnet 4.6 · Opus 4.6 · GPT 5.5

Scale

• Post-training completed in under 24 hours.
  “we were able to post-train Nemotron 3 Ultra on the Trajectory platform in under 24 hours.”article, stated
• The all-pass rate improved from 0% baseline to 5.8%.
  “Post-trained Nemotron 3 Ultra reaches 5.8%, up from 0% at baseline”article, stated
• Rubric-criteria pass rate reached 83%, up from 52% for the base model.
  “Post-trained Nemotron 3 Ultra reaches 83%, alongside GPT 5.5 (78%), Sonnet 4.6 (85%), and Opus 4.6 (85%), up from 52% for the base model.”article, stated
• The post-trained open model is reported as at least 10x cheaper to run than leading closed models at similar quality.
  “matching their quality while costing at least 10x cheaper to run.”article, stated

Alternatives considered (roads not taken)

Use frontier closed models for legal tasks

Rejected because: Closed frontier models are described as more expensive to operate on reasoning-heavy legal tasks.

Adopt each new base model with custom engineering

Rejected because: The reusable model-agnostic layer allowed the same harness, data, and recipe to be reused with no new engineering.

Judge model quality by partial rubric gains alone

Rejected because: Legal work requires all rubric criteria to be satisfied end-to-end.

Patterns

fine-tuning-adaptation (model-adaptation)

Trajectory post-trained Nemotron 3 Ultra on legal work using a reusable model-agnostic pipeline.

eval-driven-development (quality)

The model was evaluated on held-out LAB tasks, all-pass rate, rubric criteria, and per-task score distributions.

model-routing (model-operations)

The platform is model-agnostic so each new open model can be put through the same pipeline.

cost-aware-routing (cost)

The benchmark explicitly compares quality against per-token cost to find open-weight models that match frontier quality more cheaply.

Tradeoffs

Use open-weight post-training for specialized legal work.

Chose Post-trained Nemotron 3 Ultra over Relying only on leading closed frontier models.

Rationale: Post-training reached frontier-like legal quality while lowering serving cost.

Optimize for rapid model adoption.

Chose Reusable model-agnostic learning layer over Model-specific post-training engineering.

Rationale: The same harness and recipe enabled adoption in under 24 hours.

Evaluate end-to-end all-pass in addition to rubric averages.

Chose Held-out all-pass plus rubric coverage over Only average score or single-threshold improvement.

Rationale: The all-pass metric reflects the completeness standard legal work requires.

Gotchas (what bit them)

Partial legal-task success is not enough.

Root cause: Legal tasks require satisfying every rubric criterion, not only improving average score.

Resolution: Use all-pass rate on held-out LAB tasks as the headline metric.

Closed frontier quality can be cost-prohibitive for production legal work.

Root cause: Reasoning-heavy, long-horizon tasks make per-token pricing matter.

Resolution: Post-train open-weight models and evaluate quality against cost.

New open models create operational churn.

Root cause: Each new model can reset adoption work if training infrastructure is tightly coupled to a base model.

Resolution: Decouple the learning layer from base models so each model can use the same post-training pipeline.

Evaluation

Trajectory evaluated post-trained Nemotron 3 Ultra on held-out Harvey LAB tasks using all-pass rate, rubric-criteria pass rate, per-task score distributions, capability gains, and quality-versus-cost comparison against closed frontier models.

Metrics: All-pass rate · Rubric criteria passed · Per-task score distribution · Top capability gains · Quality against cost · Price per token

Outcomes

• Nemotron 3 Ultra was post-trained in under 24 hours.
• All-pass rate improved from 0% to 5.8%.
• Rubric-criteria pass rate improved from 52% to 83%.
• The model landed in the same band as leading closed models at a fraction of the running cost.

The source reports under-24-hour post-training, 5.8% all-pass, 83% rubric pass rate, 0% baseline all-pass, 52% baseline rubric pass rate, +52 point Immigration capability lift, and at least 10x cheaper runtime versus leading closed models.