Autonomous Software Development Using Multi-Model LLM System with Advanced Planning and Tool Integration

factory.ai · Tech · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Autonomous software engineering agents

Factory.ai built Code Droid to automate real software engineering tasks from natural-language tickets while preserving reliability, safety, explainability, and production fit across diverse codebases.

Constraints:

• coding tasks require multi-step reasoning, planning, tool use, and codebase understanding
• model capabilities vary by subtask
• agents need retrieval over codebases without entering with zero knowledge
• autonomous code edits must be traceable, reversible, and security-checked
• public benchmarks underrepresent migrations, feature work, refactoring, and customer-specific tasks

Architecture

Code Droid combines planning/task decomposition, tool-grounded execution, HyperCode/ByteRank codebase retrieval, multi-model sampling, tests for candidate validation, sandboxed execution, explainability logs, DroidShield static analysis, and internal Crucible benchmarks. The system generates candidate solution trajectories across different models, validates them with tests and generated tests, and selects robust patches while keeping actions auditable and reversible.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  natural-language-ticket[Natural-language ticket] --> code-droid[Code Droid]
  code-droid --> planning-engine[Planning engine]
  code-droid --> hypercode[HyperCode]
  hypercode --> byterank[ByteRank]
  byterank --> code-droid
  code-droid --> developer-tools[Developer tools]
  developer-tools --> sandbox[Sandbox]
  code-droid --> multi-model-sampler[Multi-model sampler]
  multi-model-sampler --> test-validator[Test validator]
  test-validator --> code-droid
  sandbox --> droidshield[DroidShield]
  droidshield --> code-droid
  code-droid --> explainability-logs[Explainability logs]
  explainability-logs --> human-developer[Human developer]
  code-droid --> crucible-benchmark[Crucible benchmark]
  crucible-benchmark --> human-developer

Components

Data flow

A natural-language ticket is given to Code Droid. HyperCode constructs a multi-resolution codebase representation and ByteRank retrieves relevant context. The Droid decomposes the problem, uses developer tools, samples candidate trajectories across models, validates candidates using existing/self-generated tests, and produces patches inside a sandbox. Evaluation flows through SWE-bench, Crucible, audit logs, DroidShield, and compliance controls.

Why this architecture

Factory uses a multi-part architecture because autonomous coding needs more than a strong model: it needs planning, codebase retrieval, realistic tools, multi-model diversity, testing, safety isolation, auditability, and benchmarks that match customer work. Multi-model sampling and validation address model variability, while HyperCode/ByteRank address codebase context limits.

Models used: Anthropic models · OpenAI models · Code Droid · DroidShield

Scale

• Code Droid achieved 19.27% on SWE-bench Full.
  “We provide an analysis of its state-of-the-art performance on SWE-bench, where we achieve 19.27% on SWE-bench Full and 31.67% on SWE-bench Lite.”article, stated
• SWE-bench Full contains 2,294 issues and pull requests from twelve Python projects.
  “It is composed of 2,294 issues and pull requests gathered from twelve popular open-source Python projects. The AI system is given the problem statement extracted from the GitHub issue and the repository itself and is expected to generate a git patch to resolve the issue.”article, stated
• SWE-bench Lite includes 300 problems and Code Droid reached 42.67% pass@6.
  “This benchmark includes 300 problems, focusing on evaluating the AI system’s ability to solve more straightforward software engineering tasks efficiently.”article, stated

  “The pass rates improved with multiple attempts - specifically, Code Droid achieved a pass rate of 37.67% pass@2 and further improved to 42.67% for its pass@6.”article, stated
• Patch runtime ranged from 5-20 minutes on average to 136 minutes in an extreme case.
  “On average, Code Droid took between 5 to 20 minutes to generate a single patch, with the system producing up to three patches per task before selecting the most promising one.”article, stated

  “The most extreme case saw Code Droid taking 136 minutes to generate a patch. In terms of token usage, the runs varied significantly, with some consuming as many as 13 million tokens per patch.”article, stated
• Some runs consumed as many as 13 million tokens per patch.
  “some consuming as many as 13 million tokens per patch. However, the average token usage remained under 2 million tokens per patch.”article, stated

Alternatives considered (roads not taken)

Use one model for all coding subtasks

Rejected because: Factory found model capabilities are task-dependent and benefits from diverse candidate trajectories.

Enter each codebase with zero knowledge

Rejected because: Codebase-scale tasks require explicit and implicit representations before retrieval.

Trust public benchmarks as the only evaluation

Rejected because: SWE-bench overrepresents debugging tasks and misses many enterprise software-engineering tasks.

Run autonomous edits directly in main environments

Rejected because: The Droid is isolated to prevent destabilization and protect data.

Patterns

model-routing (model-ops)

Code Droid routes different subtasks to different Anthropic/OpenAI models based on task-dependent strengths.

rag-retrieval (code-context)

HyperCode and ByteRank retrieve codebase information from multi-resolution representations.

tool-use-orchestration (developer-tools)

The Droid uses version control, editors, debugging tools, linters, static analyzers, and tests.

eval-driven-development (evaluation)

SWE-bench, Crucible, failure analysis, and continuous benchmark updates guide Droid improvement.

safety-policy-enforcement (safety)

DroidShield and sandboxing enforce safety before code is committed.

Tradeoffs

Generate and validate multiple trajectories

Chose sampling across models with test-based selection over single deterministic patch generation.

Rationale: Diverse candidate solutions improve robustness but increase runtime and token cost.

Prioritize safety and explainability

Chose sandboxing, audit trails, logs, DroidShield, and certifications over unconstrained autonomous code edits.

Rationale: Enterprise deployment requires trust, reversibility, security, and compliance.

Build internal evaluations

Chose Crucible customer-centric benchmark suite over only optimizing for SWE-bench.

Rationale: Real customer work spans tasks that SWE-bench does not adequately represent.

Gotchas (what bit them)

Codebase context retrieval remains a major failure mode.

Root cause: The agent sometimes misses or misprioritizes target files.

Resolution: Factory analyzes failure modes and improves retrieval, prioritization, and editing selection.

Benchmark contamination and exact-patch overlap require audit.

Root cause: Language models may have seen repositories or similar patches in training data.

Resolution: Factory revoked internet access, compared patches to oracle patches, and manually reviewed close matches.

Runtime and token usage vary widely.

Root cause: Autonomous coding trajectories can be long and produce multiple candidate patches.

Resolution: Factory reports runtime/token variability as a production economics concern.

Public benchmarks do not cover the whole product target.

Root cause: SWE-bench is mostly debugging-style work, while Factory targets migrations, features, refactors, integrations, and reviews.

Resolution: Factory created Crucible for customer-centric and evolving evaluation scenarios.

Evaluation

Factory evaluates Code Droid with SWE-bench Full/Lite, pass@k attempts, failure-mode analysis, runtime/token metrics, oracle-patch contamination checks, and the Crucible internal benchmark for real customer workflows.

Metrics: 19.27% SWE-bench Full · 31.67% SWE-bench Lite first attempt · 37.67% pass@2 and 42.67% pass@6 · 8% target-file missing, 8% target-file not top-5, 6% not selected for editing · 5-20 minutes average patch runtime and up to 13 million tokens per patch

Outcomes

• Code Droid outperformed listed competitors on Factory's reported SWE-bench subset and achieved published SWE-bench Full/Lite scores.
• Failure-mode analysis revealed retrieval and edit-selection bottlenecks.
• Factory created Crucible to evaluate tasks closer to customer production work.
• The system includes enterprise safety, audit, explainability, and compliance controls.

The source provides substantial metrics: 2,294 SWE-bench issues, 300 Lite problems, 19.27% and 31.67% headline scores, pass@6 improvement to 42.67%, 5-20 minute average patch time, a 136-minute extreme, and up to 13 million tokens per patch.