Fine-Tuning Financial Document Filtering with Expert Judgment

bridgewater_aia_labs_/_thinking_machines · Finance · 2026 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: investment research information triage

Bridgewater AIA Labs and Thinking Machines wanted to automate financial document filtering tasks that were routine for experienced investors but difficult for frontier LLMs, because the relevant judgment was expert taste embedded in daily investment workflows rather than easily written prompt instructions.

Constraints:

• the target tasks depended on investor judgment that was hard to articulate directly
• frontier models performed below the trust threshold even after prompt engineering
• training labels had to reflect expert investor taste rather than non-expert annotations
• expert labeling was expensive, so only the most contested examples could be routed back to experts
• the resulting model needed to improve accuracy while reducing inference cost per task

Architecture

The team built a custom fine-tuned financial triage model around six investor workflow tasks, expert annotations, and a Qwen3-235B base model trained on Tinker. They first measured frontier-model and prompt-engineering limits, cleaned non-expert labels by routing model-disagreement cases to expert investors, then improved the training recipe with interleaved task batching, CISPO loss with asymmetric clipping, and on-policy distillation that promoted validation-improving checkpoints to the teacher.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  financial-document-corpus[Financial document corpus]
  investor-task-definitions[Investor task definitions]
  expert-annotation-workflow[Expert annotation workflow]
  disagreement-routing-scheme[Disagreement routing scheme]
  training-dataset[Training dataset]
  tinker-training-platform[Tinker training platform]
  qwen3-235b-financial-judgment-model[Qwen3-235B financial judgment model]
  validation-promoted-teacher[Validation-promoted teacher]
  held-out-evaluation-suite[Held-out evaluation suite]
  financial-document-corpus --> investor-task-definitions
  financial-document-corpus --> expert-annotation-workflow
  investor-task-definitions --> held-out-evaluation-suite
  expert-annotation-workflow --> disagreement-routing-scheme
  disagreement-routing-scheme --> training-dataset
  expert-annotation-workflow --> training-dataset
  training-dataset --> tinker-training-platform
  training-dataset --> held-out-evaluation-suite
  tinker-training-platform --> qwen3-235b-financial-judgment-model
  qwen3-235b-financial-judgment-model --> validation-promoted-teacher
  validation-promoted-teacher --> qwen3-235b-financial-judgment-model
  qwen3-235b-financial-judgment-model --> held-out-evaluation-suite

Components

Data flow

Financial documents and task definitions are labeled for six investor filtering tasks, with contested non-expert examples sent to Bridgewater experts for reevaluation. The cleaned dataset trains a Qwen3-235B model on Tinker. Training iterates through interleaved task batches, CISPO loss, and on-policy distillation, while validation accuracy determines when a checkpoint becomes the teacher. The final trained model is evaluated against held-out tasks and compared to frontier models on accuracy, F1, and cost.

Why this architecture

The team chose custom fine-tuning because expert investment judgment was difficult to encode in prompts, frontier models remained below an 80% trust threshold, and high-quality proprietary annotations let a domain-specific model outperform frontier models at lower cost.

Models used: Qwen3-235B · Gemini variants · Claude variants · GPT variants

Scale

• The public evaluation covered six information filtering tasks from investor workflows.
  “We evaluated models on six information filtering tasks drawn from investors’ daily workflows.”article, stated
• Simple prompts gave frontier models around 50% average accuracy on the six tasks.
  “Variants of Gemini, Claude, and GPT averaged a mere ~50% accuracy when given a prompt that simply states each of the six tasks to perform.”article, stated
• Manual and automatic prompt engineering improved frontier model accuracy only into the mid-70s.
  “These changes boosted their accuracy from a coin flip to the mid-70s.”article, stated
• The trained model improved average accuracy from 78.2% to 84.7%.
  “Our trained model improves average accuracy from 78.2% to 84.7%, meaning the trained model makes 29.8% fewer mistakes than the best frontier model we evaluated.”article, stated
• The custom model reduced inference costs per task by 13.8x.
  “Our trained model is also vastly cheaper due to its smaller size: a 13.8x reduction in inference costs per task.”article, stated

Alternatives considered (roads not taken)

frontier models with simple task prompts

Rejected because: they averaged only about 50% accuracy on the financial tasks.

stronger manual and automatic prompt engineering

Rejected because: the best prompts still missed the 80% accuracy threshold investors required for daily workflow trust.

train directly on non-expert vendor labels

Rejected because: models trained on those labels performed poorly because many labels were wrong.

distill from a frozen base-model teacher

Rejected because: promoting only validation-improving checkpoints as teachers produced better results than a frozen teacher.

Patterns

fine-tuning-adaptation (model-operations)

The team fine-tuned Qwen3-235B on proprietary investor-labeled financial tasks so the model could learn judgment that prompts could not fully express.

human-in-the-loop-gate (human-oversight)

A disagreement-routing scheme sent only contested examples back to expert investors, preserving scarce expert attention for the highest-value labels.

eval-driven-development (evaluation)

Model changes were assessed with accuracy, F1, held-out test sets, validation-high checkpoint promotion, and leave-one-out ablations of the training recipe.

model-distillation (model-operations)

On-policy distillation regularized the student against a teacher distribution, with the teacher updated only from validation-improving checkpoints.

cost-aware-routing (serving)

The custom smaller model was justified not only by accuracy but by a large inference-cost reduction compared with frontier models.

Tradeoffs

Fine-tune a model rather than keep elaborating prompts.

Chose example-driven custom model training over static expert-written prompts.

Rationale: Prompts could not capture the tacit investor judgment needed for reliable document triage.

Route contested examples to experts instead of relabeling everything with experts.

Chose selective expert verification over full expert relabeling or trusting non-expert labels.

Rationale: Expert labels were costly, but disagreement cases concentrated likely hard or wrong examples.

Use Tinker rather than managing training infrastructure directly.

Chose managed experimentation on Tinker over operating GPU infrastructure for the training loop.

Rationale: The team wanted rapid iteration on the training recipe without infrastructure management becoming the bottleneck.

Promote teachers by validation high-water mark.

Chose validation-gated teacher updates over distilling against a frozen teacher or every checkpoint.

Rationale: The validation gate avoided distilling toward weaker checkpoints and improved performance over the frozen-teacher variant.

Gotchas (what bit them)

Frontier models were weak at tasks that investors considered simple.

Root cause: The relevant judgments depended on tacit financial taste and context rather than surface task instructions.

Resolution: The team moved from prompting toward custom fine-tuning on expert-labeled proprietary data.

Prompt improvements plateaued below the trust threshold.

Root cause: Explicit instructions could convey only the part of expert intuition that could be written down.

Resolution: The final approach used fine-tuning so the model could learn from examples rather than static prompts alone.

Non-expert training labels degraded model performance.

Root cause: The dataset labels were often wrong when judged against expert investor taste.

Resolution: A model-disagreement verification scheme routed suspicious examples to experts for reevaluation.

Distillation could reinforce a weaker teacher if checkpoint promotion were unconditional.

Root cause: On-policy distillation depends on the teacher distribution, so a weaker teacher would regularize the student toward poorer behavior.

Resolution: The teacher was updated every 20 steps only when validation accuracy reached a new high.

Evaluation

The team compared frontier models, prompt-engineered baselines, Qwen fine-tuning recipes, and ablations on six investor workflow tasks using average accuracy, positive-class F1 for classification tasks, held-out test sets, and inference cost per task.

Metrics: average accuracy across all six tasks · positive-class F1 averaged across three classification tasks · frontier models around 50% accuracy with simple prompts · best prompt-engineered frontier models under 80% accuracy · Qwen + GRPO at 73.48% average accuracy · final recipe at 84.66% average accuracy and 92.99% average positive F1 · 13.8x reduction in inference costs per task

Outcomes

• The custom model improved average accuracy from 78.2% to 84.7% over the best frontier model evaluated.
• The trained model made 29.8% fewer mistakes than the best frontier baseline.
• The model reduced inference costs per task by 13.8x.
• Interleaved batching improved accuracy by 12.1% over fully mixed batches.
• CISPO loss with asymmetric clipping improved accuracy by 10.1% over the importance-sampling baseline.
• Validation-gated on-policy distillation improved 3.1% over a frozen base-model teacher.

The public release covers six financial filtering tasks, but the authors state that similar internal tasks showed the same frontier-model underperformance pattern and that they expect more custom models for organizational needs.