Training Agentic Models with Reinforcement Learning for Production Deployment

kimi_/cursor/_chroma · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Reinforcement learning for production agents

Kimi, Cursor, and Chroma needed agentic models that work in real production harnesses, manage long context, avoid reward hacking, and outperform benchmark-only training.

Constraints:

Training environments had to match production deployments.
Long agentic sessions created context-window management problems.
Reward functions could create degenerate behaviors.
Public benchmarks did not capture real-world tasks.
Multi-agent RL created credit-assignment and cost-measurement challenges.

Architecture

The case compares three RL-trained agent systems. Kimi K2.5 uses an Agent Swarm with a trainable orchestrator and frozen sub-agents; Cursor Composer 2 trains from production coding traffic in a real-time deployment loop; Chroma Context-1 trains a 20B retrieval model for self-editing search. Across systems, teams train inside production-like harnesses, use outcome rewards plus generative reward models, run large asynchronous rollouts, and build domain-specific evaluations.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production_equivalent_harness["Production-equivalent harness"] --> rl_rollout_system["RL rollout system"]
  rl_rollout_system --> reward_models["Reward models"]
  reward_models --> trainable_agent_model["Trainable agent model"]
  trainable_agent_model --> production_equivalent_harness
  domain_specific_benchmarks["Domain-specific benchmarks"] --> production_equivalent_harness
  kimi_orchestrator["Kimi orchestrator"] --> frozen_kimi_sub_agents["Frozen Kimi sub-agents"]
  kimi_orchestrator --> rl_rollout_system

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Production-equivalent harness`	Runs agent tasks in environments that resemble deployment.	`RL rollout system`, `Domain-specific benchmarks`
`RL rollout system`	Collects trajectories from agent attempts at production-like tasks.	`Production-equivalent harness`, `Reward models`, `Trainable agent model`
`Reward models`	Score outcomes and discourage degenerate agent behaviors.	`RL rollout system`, `Trainable agent model`
`Trainable agent model`	Learns production task behavior through reinforcement learning.	`RL rollout system`, `Production-equivalent harness`
`Domain-specific benchmarks`	Measure real task progress beyond public benchmark coverage.	`Production-equivalent harness`
`Kimi orchestrator`	Creates, assigns, and aggregates sub-agent work in Agent Swarm.	`Frozen Kimi sub-agents`, `RL rollout system`
`Frozen Kimi sub-agents`	Execute subtasks as environmental observations during orchestrator training.	`Kimi orchestrator`

Data flow

Tasks run inside production-equivalent agent harnesses. Rollouts generate trajectories, reward functions score task success and failure modes, RL updates train the model or orchestrator, and domain-specific benchmarks plus production traffic close the loop for deployment.

3 supporting quotes

“Common solutions include training inside production harnesses, using outcome-based rewards augmented with generative reward models, running asynchronous large-scale rollouts”summary, stated

“Each team independently concluded that training must occur inside production-equivalent environments”summary, stated

“that reward engineering requires iterative refinement to prevent reward hacking.”summary, stated

Why this architecture

The shared architecture follows from a production gap: public benchmarks and offline training do not teach models to operate in real agent harnesses, so the systems train on production-equivalent rollouts and domain-specific rewards.

“traditional public benchmarks inadequately measure real-world performance”summary, stated

“The convergence suggests these patterns represent emerging best practices for deploying agentic LLMs rather than company-specific choices.”summary, stated

Models used: Kimi K2.5 · Cursor Composer 2 · Chroma Context-1 · Generative reward models

Scale

Kimi K2.5 is a 1T-parameter MoE with 32B active parameters.
“Kimi K2.5 is a 1 trillion parameter mixture-of-experts model with 32B active parameters”summary, stated
Kimi achieved 78.4% BrowseComp accuracy and a 4.5x latency reduction.
“Kimi K2.5 introduces Agent Swarm for parallel task decomposition, achieving 78.4% accuracy on BrowseComp with 4.5× latency reduction.”summary, stated
Cursor Composer 2 used a five-hour deployment cycle and trained on tasks with median 181-line changes.
“Cursor Composer 2 implements real-time RL from production traffic with a five-hour deployment cycle, training on tasks with median 181-line changes.”summary, stated
Chroma Context-1 is a 20B model matching frontier-scale performance at 10x speed.
“Chroma Context-1 develops self-editing search capabilities in a 20B parameter model that matches frontier-scale performance at 10× speed.”summary, stated

Alternatives considered (roads not taken)

Train on public benchmarks alone

Rejected because: The teams found public benchmarks inadequate for real production performance.

“traditional public benchmarks inadequately measure real-world performance”summary, stated

Train all Kimi swarm agents jointly

Rejected because: Freezing sub-agents solved credit assignment for orchestrator training.

“By freezing sub-agents and treating their outputs as environmental observations, only the orchestrator's coordination logic receives gradient updates.”summary, stated

Reward parallelism directly forever

Rejected because: Auxiliary rewards were annealed so final policy optimized task performance rather than spurious parallelism.

“The auxiliary reward coefficients (parallelism and finish) are annealed to zero over training, ensuring the final policy optimizes solely for performance”summary, stated

Patterns

`eval-driven-development` (agent training)

The teams built domain-specific evaluation benchmarks because public benchmarks did not represent real production tasks.

“Common solutions include training inside production harnesses, using outcome-based rewards augmented with generative reward models, running asynchronous large-scale rollouts, and building domain-specific evaluation benchmarks.”summary, stated

`multi-agent-orchestration` (parallel agents)

Kimi K2.5 uses a trainable orchestrator to create sub-agents, assign work, and aggregate results.

“the model dynamically decomposes tasks into parallel subtasks executed by independent sub-agents.”summary, stated

`context-engineering` (long sessions)

The systems treat long-context management during agent sessions as a central production training problem.

“All three teams face similar challenges: managing context windows during long agentic sessions, bridging the gap between training environments and production deployments, and designing reward functions that avoid degenerate behaviors.”summary, stated

`llm-as-judge` (reward modeling)

Outcome rewards are augmented with generative reward models for agentic training.

Tradeoffs

Measure Kimi parallel work by critical steps.

Chose Critical-path cost over Total summed steps across all agents.

Rationale: Critical steps reward balanced parallel execution rather than penalizing concurrency.

“Traditional step counting sums total steps across all agents, which incorrectly penalizes parallel execution. Critical steps instead measure the longest execution chain, analogous to critical path analysis in distributed systems.”summary, stated

Freeze Kimi sub-agents during orchestrator RL.

Chose Train the orchestrator only over Jointly optimize orchestrator and sub-agents.

Rationale: This makes the coordination credit assignment problem tractable.

“By freezing sub-agents and treating their outputs as environmental observations, only the orchestrator's coordination logic receives gradient updates.”summary, stated

Gotchas (what bit them)

Training outside production harnesses creates deployment mismatch.

Root cause: Benchmarks and synthetic environments miss real workflow behavior.

Resolution: Train inside production-equivalent environments and use production traffic where appropriate.

Reward functions can induce degenerate behavior.

Root cause: Agents exploit poorly shaped rewards instead of solving tasks.

Resolution: Iteratively refine reward engineering and anneal auxiliary signals.

“designing reward functions that avoid degenerate behaviors.”summary, stated

“reward engineering requires iterative refinement to prevent reward hacking.”summary, stated

Parallel agent training creates credit assignment ambiguity.

Root cause: A correct final answer may come from a good orchestrator or a sub-agent compensating for poor decomposition.

Resolution: Freeze sub-agents and optimize only orchestrator coordination.

“when a final answer is correct, it's ambiguous whether the orchestrator decomposed the task well or a sub-agent compensated for poor decomposition”summary, stated

Evaluation

The systems use production-equivalent harnesses, domain-specific benchmarks, BrowseComp accuracy, production traffic, and task-change statistics rather than relying only on public benchmarks.

Metrics: Kimi BrowseComp accuracy of 78.4%. · Kimi latency reduction of 4.5x. · Cursor five-hour deployment cycle. · Cursor median 181-line task changes. · Chroma 20B model at 10x speed relative to frontier-scale performance.

“Kimi K2.5 introduces Agent Swarm for parallel task decomposition, achieving 78.4% accuracy on BrowseComp with 4.5× latency reduction.”summary, stated

“Cursor Composer 2 implements real-time RL from production traffic with a five-hour deployment cycle, training on tasks with median 181-line changes.”summary, stated

Outcomes

Three teams converged on production-harness RL for agentic models.
Kimi demonstrated parallel agent orchestration with strong BrowseComp and latency results.
Cursor trained coding agents from production traffic.
Chroma trained a smaller retrieval model with frontier-like performance at higher speed.

The case cites Kimi's 1T/32B-active MoE, Cursor's median 181-line production tasks, and Chroma's 20B retrieval model.

“This case study examines three production LLM systems—Kimi K2.5, Cursor Composer 2, and Chroma Context-1—that use reinforcement learning”summary, stated