Using RL to Make a 4B Parameter Model Outperform a 235B Parameter Model on Financial Analysis Tool Use

snorkel · Finance · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Financial analysis tool-use model training

Snorkel and UC Berkeley needed to improve enterprise financial-analysis tool use without defaulting to much larger, more expensive reasoning models.

Constraints:

• Financial and healthcare deployments often need lower cost, on-premise control, and data sovereignty.
• The target behavior was disciplined tool use, not general knowledge acquisition.
• The evaluation environment needed to be self-contained and reproducible.
• Training data required expert curation and answer verification.
• The comparison focused on a constrained structured-tool environment rather than arbitrary open-ended tool use.

Architecture

Snorkel and Berkeley built a financial tool-use training loop around expert-curated data, a self-contained FinQA environment, and GRPO reinforcement learning. Experts create answerable financial-analysis tasks, FinQA exposes table-discovery, schema-inspection, and SQL tools, rubrics evaluate behavior, and a 4B base model is trained for 21 hours under $500 to inspect tools, query schemas, recover from SQL errors, and outperform a 235B reasoning model on pass-at-one.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  expert-curated-task-dataset["Expert-curated task dataset"] --> finqa-environment["FinQA environment"]
  expert-curated-task-dataset --> grpo-training-loop["GRPO training loop"]
  finqa-environment --> tool-interface["Tool interface"]
  tool-interface --> four-billion-parameter-model["Four billion parameter model"]
  four-billion-parameter-model --> grpo-training-loop
  grpo-training-loop --> evaluation-benchmarks["Evaluation benchmarks"]
  finqa-environment --> evaluation-benchmarks
  baseline-reasoning-model["Baseline reasoning model"] --> evaluation-benchmarks

Components

Data flow

Domain experts create verified financial-analysis tasks with known answers. The FinQA environment packages tables and tools for discovering available data, inspecting schema, and executing SQL. A 4B model interacts with the environment during GRPO training, receives reward from task outcomes, learns tool-discipline behaviors, and is evaluated on standard and harder reasoning benchmarks against larger model baselines.

Why this architecture

The team optimized a small model because the observed failure was tool discipline, not raw reasoning capacity. A self-contained environment and expert-verified tasks let reinforcement learning reward the behaviors that production systems need: inspect tools, inspect schemas, query correctly, observe errors, and self-correct.

Models used: 4B base model · 235B quantized reasoning model · GRPO reinforcement learning · FinQA · FinQA Reasoning

Scale

• The trained model had 4 billion parameters.
  “a 4 billion parameter model fine-tuned with reinforcement learning”summary, stated
• The baseline reasoning model had 235 billion parameters.
  “outperform a 235 billion parameter reasoning model”summary, stated
• Training completed in 21 hours.
  “The entire training run completed in 21 hours at a total cost of under $500 per run.”summary, stated
• Training cost under $500 per run.
  “applying GRPO reinforcement learning for under $500 in a 21-hour training run”summary, stated
• The standard FinQA benchmark has 290 samples.
  “the standard FinQA with 290 samples, and FinQA Reasoning with 79 more challenging samples”summary, stated
• FinQA Reasoning has 79 harder multi-table samples.
  “FinQA Reasoning with 79 more challenging samples requiring multi-table queries.”summary, stated
• FinQA Reasoning performance improved from 13.9% to 26.6%.
  “Performance on that benchmark jumped from 13.9% to 26.6%—again roughly a doubling.”summary, stated

Alternatives considered (roads not taken)

Deploy a larger reasoning model

Rejected because: The 235B model still lacked tool discipline and hallucinated after failed queries.

Train only on multi-table reasoning complexity

Rejected because: Single-table training generalized surprisingly well by teaching tool discipline.

Use an externally dependent evaluation environment

Rejected because: Financial deployments may need on-premise, data-sovereign evaluation with no remote dependencies.

Patterns

fine-tuning-adaptation (model training)

A 4B model was adapted with GRPO reinforcement learning for financial tool-use behavior.

tool-use-orchestration (tool discipline)

The model learned to discover tables, inspect schemas, execute SQL, observe errors, and self-correct.

eval-driven-development (training data)

Expert-verified tasks, benchmarks, and rubrics guide training and diagnosis of model behavior.

cost-aware-routing (model deployment)

The work shows that a smaller specialized model can beat a much larger model for a constrained tool-use task at lower serving cost.

structured-output-contract (environment design)

The self-contained environment exposes a fixed tool contract for table discovery, schema inspection, and SQL execution.

Tradeoffs

Optimize behavior instead of scale

Chose Train a 4B model with RL over Use a 235B reasoning model.

Rationale: The bottleneck was systematic tool interaction, not general reasoning capacity.

Single-table training curriculum

Chose Train on single-table questions over Mix or progressively introduce multi-table questions.

Rationale: Single-table training best taught transferable tool discipline.

Self-contained financial environment

Chose Package tools and data without external dependencies over Depend on remote services during evaluation.

Rationale: This supports reproducibility, on-prem deployment, and data sovereignty.

Gotchas (what bit them)

Bigger reasoning models can fail from poor tool discipline.

Root cause: The 235B baseline guessed table names, retried guesses, and hallucinated instead of inspecting the environment.

Resolution: RL trained the smaller model to inspect tools, query schemas, and self-correct after errors.

Training-cost claims can omit expert-data costs.

Root cause: The reported under-$500 figure covers training, not the preceding expert curation.

Resolution: Teams should budget for data generation as well as compute.

A self-contained benchmark may understate messy integration complexity.

Root cause: Real financial systems often require external APIs, legacy systems, and longer tool chains.

Resolution: Treat FinQA results as strong evidence for structured-tool tasks, not a blanket proof for all production finance workflows.

Evaluation

The team evaluated pass-at-one performance on FinQA and FinQA Reasoning, compared the fine-tuned 4B model with a 235B baseline, and used rubrics to diagnose tool-use behaviors.

Metrics: Pass-at-one performance · FinQA 290-sample benchmark · FinQA Reasoning 79-sample benchmark · 13.9% to 26.6% FinQA Reasoning improvement · Under-$500, 21-hour GRPO run

Outcomes

• A 4B RL-trained model outperformed a 235B reasoning model on the financial tool-use task.
• Pass-at-one performance roughly doubled versus the pre-training baseline.
• FinQA Reasoning performance improved from 13.9% to 26.6%.
• The 21-hour training run cost under $500, excluding expert data work.

The headline comparison is a 4B specialized model beating a 235B quantized reasoning baseline after a 21-hour GRPO run costing under $500.