Fine-Tuning LLMs for Multi-Agent Orchestration in Code Generation

cosine · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: enterprise coding agents

Cosine needed coding agents that could perform long-horizon software engineering work for regulated enterprises with large codebases, niche stacks, and deployment constraints where frontier models or hosted egress-heavy systems were not always viable.

Constraints:

• the product had to deploy in multi-tenant cloud, single-tenant VPC, and fully air-gapped on-premise hardware
• small models were required in constrained enterprise environments but struggled with long-horizon coding tasks
• GPU supply and deployment isolation constrained the viable model footprint
• agent work had to integrate with developer pull-request workflows and remain auditable
• orchestrator and worker models had to avoid policy drift, context pollution, repeated tool use, and doom loops

Architecture

Cosine built Genie as a specialized multi-agent coding system. A user submits a large coding task through web or CLI. An orchestrator model breaks the request into subtasks, calls worker agents as tools, inspects their chain-of-thought and git diffs, and either asks for follow-up work or completes the task with a reviewable pull request. Training starts with strong frontier software-engineering trajectories, distills successful trajectories into smaller orchestrator and worker models, then improves workers with supervised fine-tuning and reinforcement learning. For constrained deployments, multi-LoRA adapters let the same base model switch between orchestrator and worker roles while trajectories remain stored and auditable.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  developer-web-and-cli-surfaces[Developer web and CLI surfaces]
  orchestrator-model[Orchestrator model]
  worker-agents[Worker agents]
  sandbox-code-execution[Sandbox code execution]
  production-scaffolding-and-tools[Production scaffolding and tools]
  distillation-and-rl-training-pipeline[Distillation and RL training pipeline]
  multi-lora-adapter-runtime[Multi-LoRA adapter runtime]
  trajectory-store[Trajectory store]
  audit-playback[Audit playback]
  reviewable-pull-request[Reviewable pull request]
  developer-web-and-cli-surfaces --> orchestrator-model
  orchestrator-model --> worker-agents
  orchestrator-model --> trajectory-store
  orchestrator-model --> multi-lora-adapter-runtime
  worker-agents --> sandbox-code-execution
  worker-agents --> production-scaffolding-and-tools
  worker-agents --> trajectory-store
  production-scaffolding-and-tools --> distillation-and-rl-training-pipeline
  distillation-and-rl-training-pipeline --> orchestrator-model
  distillation-and-rl-training-pipeline --> worker-agents
  distillation-and-rl-training-pipeline --> trajectory-store
  multi-lora-adapter-runtime --> orchestrator-model
  multi-lora-adapter-runtime --> worker-agents
  trajectory-store --> audit-playback
  orchestrator-model --> reviewable-pull-request
  reviewable-pull-request --> developer-web-and-cli-surfaces

Components

Data flow

A developer task enters the web or CLI product and is interpreted by the orchestrator. The orchestrator creates subtasks, assigns them to worker agents with tool and codebase access, receives worker chain-of-thought and git diffs, judges whether the changes solve the original intent, and either spawns follow-up subtasks or returns a reviewable pull request. Training data flows from strong RL-post-trained frontier models solving software-engineering problems, through stored successful trajectories, into supervised fine-tuning datasets for smaller orchestrator and worker models, with RL and real-world merged-PR trajectories reinforcing later runs.

Why this architecture

Cosine chose multi-agent specialization because small models were the only feasible option in some regulated or air-gapped deployments, but monolithic small models struggled with long-horizon coding. Orchestration decomposed work into smaller subtasks, multi-LoRA reduced hardware footprint, and distillation/RL transferred frontier-model software-engineering behavior into deployable smaller models.

Models used: Llama 3.3 70B · gpt-oss-120b · Kimi K2 Thinking · DeepSeek V3.1 · custom frontier coding models · small specialized worker and orchestrator models

Scale

• Cosine targets large regulated enterprises with large codebases and niche stacks.
  “we build cutting edge coding agents, specifically targeted at large enterprises, often in highly regulated spaces with large code bases and occasionally with very niche stacks”transcript, stated
• The product can deploy across multi-tenant cloud, single-tenant VPC, and fully air-gapped on-premise hardware.
  “Our product is deployable in a multi-tenant cloud setup in a single tenant VPC and in some extreme cases fully on hardware air gap on premise.”transcript, stated
• Training-time/runtime scaffolding coupling reduced average trajectory length by about 35%.
  “we cut down our average trajectory length. So the number of steps that it takes to arrive at the solution by about 35%.”transcript, stated
• Cosine aimed to run the entire product on as few as about four H100 GPUs for enterprise use cases.
  “We always aim to make our entire product deployable on as few as about 4H 100s, which is generally the minimum that we recommend for an enterprise use case.”transcript, stated
• The multi-agent approach produced a 3x latency boost over a generic multi-agent framework.
  “we get a boost of three x in terms of latency relative to a multi, a generic multi-agent framework.”transcript, stated
• Multi-LoRA reduced GPU footprint by 60% for on-prem deployments.
  “thanks to the multi-LoRA approach, see a 60% reduction in GPU footprint for our on-prem deployments.”transcript, stated
• The orchestrator reduced final-code errors by about 20%.
  “the quality and the number of errors that you find in the final code drops by about 20%.”transcript, stated
• A Llama 70B multi-agent setup improved SWE-Lancer performance by 31% over the base worker alone.
  “this is on a LAMA 70B model here, so relatively small, we got a 31% increase by using a multi-agent system.”transcript, stated

Alternatives considered (roads not taken)

use one monolithic worker model

Rejected because: the base worker alone underperformed the multi-agent setting on SWE-Lancer and lacked the orchestrator's review and iteration loop.

deploy large frontier models everywhere

Rejected because: regulated and air-gapped customers often could not deploy or use those models for origin, GPU, isolation, or policy reasons.

manually create supervised fine-tuning trajectories

Rejected because: manual trajectory creation was too expensive at scale compared with distilling successful trajectories from stronger models.

skip supervised fine-tuning and go straight to RL

Rejected because: weak base models needed too much RL time to learn useful behavior from scratch.

Patterns

multi-agent-orchestration (agent-architecture)

Cosine uses an orchestrator model to decompose coding tasks, delegate to worker agents, inspect their outputs, and continue iteration until the task is complete.

tool-use-orchestration (agent-architecture)

Sub-agents are exposed to the orchestrator as tool calls, and the tool response contains worker reasoning and diffs.

model-distillation (model-operations)

Cosine uses trajectories from strong frontier coding models as supervised fine-tuning data for smaller orchestrator and worker models.

fine-tuning-adaptation (model-operations)

The worker model is first moved into a software-engineering-agent frame with supervised fine-tuning before reinforcement learning.

observability-tracing (operations)

Cosine stores orchestrator and worker trajectories so enterprises can replay agent behavior and understand why actions happened.

human-in-the-loop-gate (human-oversight)

The agent's terminal output is a reviewable pull request that fits into existing developer review workflows.

Tradeoffs

Use a multi-agent architecture instead of a single small model.

Chose orchestrator plus specialized workers over one monolithic worker model.

Rationale: The orchestrator decomposes long-horizon tasks, reviews worker outputs, and improves SWE-Lancer performance without adding GPUs.

Use multi-LoRA adapters for constrained deployments.

Chose adapter-swapped orchestrator and worker personalities on one base model over separate large models for each role.

Rationale: Adapter swapping reduces GPU footprint while preserving role specialization for on-prem environments.

Distill trajectories from stronger models.

Chose frontier-model trajectory distillation over manual trajectory creation for every small-model dataset.

Rationale: Manual trajectory construction was possible but expensive and slow, while distillation transferred stronger model abilities into smaller models.

Store and replay trajectories.

Chose fully auditable orchestrator and worker traces over opaque agent execution.

Rationale: Regulated enterprises needed to understand why an agent took each action, and stored successful trajectories also create training signal.

Use reviewable pull requests as the delivery object.

Chose developer review workflow integration over unreviewed direct code changes.

Rationale: PR output lets developers review and iterate on agent changes inside familiar software development lifecycle practices.

Gotchas (what bit them)

Small models struggled with long-horizon coding tasks in constrained enterprise environments.

Root cause: They were not always trained for long trajectories or tool calling, and some attention mechanisms could make them forget earlier actions.

Resolution: Cosine used an orchestrator to decompose tasks, keep workers iterating, and fill intelligence gaps in smaller models.

Multi-agent systems can fail catastrophically when coordination breaks down.

Root cause: Sub-agents can get stuck, or the orchestrator can become incoherent and unable to recover the whole system.

Resolution: The architecture emphasizes specialized orchestrator training, stored trajectories, and orchestrator review of worker diffs.

Independent orchestrator and worker training can produce policy drift.

Root cause: The two model roles may learn to think about and operate on problems differently.

Resolution: Cosine treats orchestrator and worker training as separate disciplines and experiments with training the whole system online.

Poor context management degrades smaller agent models.

Root cause: Context pollution makes long trajectories harder for smaller models to reason over.

Resolution: The orchestrator/worker split scopes context per task and stores trajectories for audit and reinforcement signal.

Skipping SFT before RL leaves weak models stuck for too long.

Root cause: The base model's initial software-engineering ability limits how quickly RL can discover good trajectories.

Resolution: Cosine uses SFT to put the model into a software-engineering-agent frame before online RL.

Evaluation

Cosine evaluates coding agents on SWE-Lancer, stores successful trajectories such as merged PRs as future training signal, uses verifiable rewards for software-engineering problems, and compares base-worker performance against the multi-agent setting.

Metrics: 35% reduction in average trajectory length from training/runtime scaffolding coupling · 3x latency boost versus a generic multi-agent framework · 60% GPU footprint reduction for on-prem deployments · about 20% fewer errors in final code · 31% SWE-Lancer improvement from multi-agent setup versus base worker · deployable on as few as about four H100 GPUs

Outcomes

• Cosine reports about 35% shorter average trajectories from tight coupling between production scaffolding and training.
• The multi-agent system reports a 3x latency boost over a generic multi-agent framework.
• Multi-LoRA reports a 60% GPU-footprint reduction for on-prem deployments.
• The orchestrator reports about 20% fewer final-code errors.
• A Llama 70B multi-agent setup reports a 31% SWE-Lancer improvement over the base worker alone.
• The architecture enables isolated deployments with no egress from customer VPCs for regulated industries.

The system targets highly regulated enterprise customers and constrained deployments, including cloud, VPC, and fully air-gapped on-premise hardware, with a design goal of running on as few as about four H100 GPUs.