Building an Autonomous AI Software Engineer with Multi-Turn RL and Codebase Understanding

devin · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Autonomous software engineering agents

Cognition needed Devin to work in large existing codebases where context windows, codebase structure, historical metadata, and specialized implementation domains make generic LLM coding unreliable.

Constraints:

Large codebases exceed effective reasoning context even when nominal context windows are large.
Useful codebase knowledge includes PRs, git history, team discussions, and macro architecture, not only files.
Autonomous agents need deep research before editing.
Specialized domains such as CUDA kernels need post-training beyond general frontier models.

Architecture

Cognition built DeepWiki as a real-time continuously updated interactive wiki over a codebase, combining code and metadata into macro and micro context. Devin Search uses these artifacts for deep codebase research before work. For narrow domains, Cognition trains specialized models such as Kevin 32B with multi-turn reinforcement learning and verifiable test harnesses, allowing models to improve over iterative coding attempts.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  developer-ticket[Developer Ticket] --> devin-agent[Devin Agent]
  devin-agent --> devin-search[Devin Search]
  devin-search --> deepwiki[DeepWiki]
  codebase-metadata[Codebase Metadata] --> deepwiki[DeepWiki]
  codebase[Codebase] --> deepwiki[DeepWiki]
  devin-agent --> codebase[Codebase]
  specialized-models[Specialized Models] --> devin-agent[Devin Agent]
  training-harness[Training Harness] --> specialized-models[Specialized Models]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Developer Ticket`	Task request for autonomous implementation.	`Devin Agent`
`Devin Agent`	Autonomous software engineer moving from ticket toward pull request.	`Devin Search`, `Codebase`, `Specialized Models`
`DeepWiki`	Continuously updated interactive codebase wiki with docs, diagrams, and Q&A.	`Devin Search`, `Codebase Metadata`
`Codebase Metadata`	PRs, git history, team discussions, and macro context.	`DeepWiki`
`Devin Search`	Deep codebase research layer that retrieves macro and micro context.	`Devin Agent`, `DeepWiki`
`Codebase`	Source project being understood and modified.	`DeepWiki`, `Devin Agent`
`Specialized Models`	Domain-trained models such as Kevin 32B for CUDA kernels.	`Devin Agent`, `Training Harness`
`Training Harness`	Parses, validates, tests, and scores multi-turn model attempts.	`Specialized Models`

Data flow

A codebase is indexed into DeepWiki with documentation, diagrams, metadata, and question-answering. Devin Search retrieves macro and micro context from that representation before the agent works from ticket to pull request. Specialized model training uses generated attempts, parsing/validity checks, performance tests, and multi-turn RL to improve domain-specific coding.

“called DeepWiki. DeepWiki is a real time continually updated index of your codebase published as an interactive wiki”transcript, stated

“do. Devon search gives Devon context. It's an essential part under the hood of how Devon, the autonomous AI agent, can actually make useful changes inside larger teamwide code bases. Once you get”transcript, stated

Why this architecture

Cognition treats codebase understanding as more than stuffing files into context. DeepWiki gives macro structure and metadata, Devin Search provides focused research, and narrow-domain RL makes specialized coding tasks verifiable and optimizable.

“And codebase understanding is pretty hard. Um, LLM are amazing at so many things, but they have limited context windows.”transcript, implied

“Is there documentation? The git commit history. All of this metadata is a really useful source for building these high context wikis. Once you have those concepts, then you can connect them to the code. So what are the connections between the various code files and the”transcript, stated

Models used: Devin · Kevin 32B · multi-turn GPO/RL-trained models

Scale

Thousands of codebases linked DeepWiki as official documentation.
“is. And we've had, you know, thousands of code bases start linking to DeepWiki as part of their official documentation. So definitely check this out if you're working on open source code yourself. How does this work under the hood? We just said that LMS are really”transcript, stated
Kevin 32B reached 91% correctness on the focused kernel benchmark section.
“model? And Kevin 32B is getting 91% correct on this se section of the kernel bench benchmark that we focused on. And you can see that compared to even 04 mini or 03 this is a significant improvement. Um this is a narrow domain where high compute RL lets you”transcript, stated
DeepWiki is described as a real-time Confluence-like codebase page.
“almost like a real time confluence page with documentation diagrams and the ability to ask questions about your code.”transcript, stated

Alternatives considered (roads not taken)

Rely on large context windows alone.

Rejected because: Effective reasoning context is lower than advertised and may not capture codebase structure.

“fits inside the context window, uh, the effective context window is often a lot lower than the advertised context window. We have a series of internal benchmarks that measures effective reasoning capacity across a context. And we find very consistently that the”transcript, stated

Use only source code as context.

Rejected because: PRs, git history, and discussions carry important macro codebase context.

“itself. There's extremely rich information in the metadata around the source code. For example, was that source code added in as part of a pull request? Well, which member of the team added that pull request? What else have they contributed to? Was there”transcript, stated

Use frontier general models for CUDA kernels.

Rejected because: A specialized 32B model could outperform in a narrow, verifiable domain.

“32B. Kevin is uh, uh, outperforms many state-of-the-art foundation models on the narrow domain of writing CUDA kernels. Raise your hand if you've ever heard of a CUDA kernel. All right, we have an audience that's very familiar with the the underfittings of ML.”transcript, stated

Patterns

`agent-memory` (codebase-context)

DeepWiki acts as a durable, queryable memory of codebase structure and metadata.

“real time continually updated index of your codebase”transcript, stated

`retrieval-ranking` (code-search)

Devin Search retrieves macro and micro codebase context before autonomous work.

“that context again includes both source files, but also wiki pages.”transcript, stated

`fine-tuning-adaptation` (model-training)

Kevin 32B is specialized for CUDA kernels through multi-turn training and benchmark feedback.

“And so we use multi-turn gpo.”transcript, stated

Tradeoffs

Codebase representation

Chose DeepWiki plus Devin Search over Raw-file context stuffing.

Rationale: The wiki gives both macro architecture and searchable detailed context.

“using both the micro code, the individual files, but also the macro context it has from this wiki data structure.”transcript, stated

Specialization strategy

Chose Narrow domain post-training over General model only.

Rationale: Verifiable domains such as CUDA kernels reward domain-specific RL.

“the domain of writing CUDA kernels is extremely”transcript, stated

Gotchas (what bit them)

Nominal context is not enough for autonomous coding.

Root cause: Effective reasoning over a large codebase is lower than the advertised context window.

Resolution: Build codebase-specific macro/micro context through DeepWiki and search.

“advertised context window is much higher than the effective reasoning context”transcript, stated

Codebase knowledge lives outside files.

Root cause: Design principles and historical decisions appear in PRs, git history, and discussions.

Resolution: Include metadata in the codebase knowledge layer.

“you don't just get them from the source code itself.”transcript, stated

Evaluation

Cognition evaluates codebase-understanding utility through adoption of DeepWiki and evaluates specialized models with parse, validity, correctness, and performance test harnesses, including the KernelBench-focused Kevin 32B result.

Metrics: DeepWiki adoption · CUDA kernel correctness · Parse validity · Benchmark performance · Ticket-to-PR completion

“is it actually valid CUDA?”transcript, stated

Outcomes

DeepWiki became official documentation for thousands of codebases.
Devin Search uses DeepWiki context for autonomous codebase research.
Kevin 32B achieved 91% correctness on the focused CUDA kernel benchmark section.

The source reports thousands of DeepWiki-linked codebases and 91% Kevin 32B correctness on a KernelBench section.

“Kevin 32B is getting 91% correct”transcript, stated