Large-Scale Analysis of AI Coding Tool Adoption and Productivity Impact Across 1,000 Companies

jellyfish · Tech · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise AI coding-tool adoption measurement

Jellyfish wanted to measure what AI coding-tool adoption actually does across real software organizations, separating interactive coding assistants from autonomous agents and explaining why some high-adoption teams do not see productivity gains. The main production question was not model accuracy, but workflow-level observability across tool usage, source control, task systems, architecture, productivity, and quality.

Constraints:

• the study spans tool usage, source-control, PR review, and task-management systems
• AI coding-tool adoption must be measured as developer behavior, not only lines of code
• autonomous-agent usage needs to be separated from interactive coding assistants
• productivity metrics vary with work scoping and code architecture
• quality concerns require bug creation, bug resolution, and revert metrics
• cross-repository context is difficult for humans, tools, and agents
• absolute PR counts are misleading across different architectures

Architecture

Jellyfish built a multi-source analytics view over AI coding usage and software-delivery telemetry. The dataset joins AI tool interactions from Copilot, Cursor, Claude Code, autonomous agents such as Devon and Codeex, PR review bots, GitHub/source-control data, and task-management context from Linear and Jira. It then computes adoption rate, PR throughput, cycle time, PR size, bug/revert metrics, and an active-repos-per-engineer architecture metric. The analysis segments companies by architecture regime to show that centralized and balanced architectures capture much larger AI productivity gains than highly distributed architectures, where context is fragmented across repositories.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ai["AI tool usage feeds"] --> warehouse["Engineering analytics warehouse"]
  agents["Autonomous agent feeds"] --> warehouse
  source["Source control data"] --> warehouse
  tasks["Task management data"] --> warehouse
  warehouse --> adoption["Adoption metric"]
  adoption --> prod["Productivity metrics"]
  adoption --> quality["Quality metrics"]
  warehouse --> arch["Architecture segmentation"]
  arch --> prod
  arch --> context["Context engineering analysis"]

Components

Data flow

Jellyfish ingests AI tool usage, autonomous-agent interactions, PR review bot signals, GitHub/source-control data, and Linear/Jira task data. It aggregates developer AI adoption as the fraction of coding time using AI tools and averages that across a company. Weekly company snapshots combine adoption rate with PR throughput, cycle time, PR size, bug tickets, reverts, bug resolution, and active repositories per engineer. The architecture segmentation then compares productivity trends for centralized, balanced, distributed, and highly distributed codebases.

Why this architecture

Jellyfish uses a multi-source analytics architecture because AI coding-tool impact cannot be inferred from model metrics or lines of code alone. Adoption must be tied to actual PR behavior, cycle time, quality, task context, and architecture. Active repositories per engineer is used to expose whether context fragmentation across repos is blocking AI tools and agents from translating adoption into productivity.

Models used: GitHub Copilot · Cursor · Claude Code · Devon · Codeex · PR review bots · interactive AI coding tools · autonomous coding agents

Scale

• The dataset contains about 20 million pull requests.
  “our customers comprises about 20 million poll requests.”transcript, stated
• The study covers about 200,000 developers.
  “these were written emerged by about 200,000 uh developers”transcript, stated
• The study spans roughly 1,000 companies.
  “from around a thousand companies.”transcript, stated
• The time window starts in June 2024 and runs to the presentation's present.
  “results that span from June 2024 to the present.”transcript, stated
• Median company AI adoption grew from about 22% to close to 90%.
  “median adoption rate was around 22%. So, uh, median company developers are using AI 22% of the time that they code.”transcript, stated

  “we're seeing median adoption rates close to 90%.”transcript, stated
• Only about 44% of companies had done anything with autonomous agents in the previous three months.
  “we only see about 44% of companies have done anything with autonomous agents at all in the past 3 months.”transcript, stated
• Autonomous-agent work was less than 2% of millions of merged PRs in that window.
  “it all amounts to less than uh 2% of the millions of PRs that were merged over that time frame.”transcript, stated
• Full AI adoption correlates with about 2x PR throughput on average.
  “The average trend here is about a 2x change as you go from zero to full adoption.”transcript, stated
• Cycle time decreased by 24% from 0% to 100% adoption.
  “the average trend is a 24% decrease in cycle times as you go from 0% to 100% adoption of AI coding tools.”transcript, stated
• PRs were 18% larger on average at full adoption.
  “teams that have fully adopted AI coding tools are pushing PRs that are 18% larger in terms of net lines of code added.”transcript, stated

Alternatives considered (roads not taken)

Use AI-generated lines of code as the primary adoption metric

Rejected because: The speaker says developer adoption better captures behavior change and correlates more directly with productivity outcomes.

Assume autonomous coding agents are already the main production driver

Rejected because: The data shows autonomous agents are mostly trials and less than 2% of merged PRs.

Compare absolute PR counts across organizations

Rejected because: Architecture changes the natural number of PRs needed, so the speaker recommends tracking change in throughput instead.

Assume microservices or highly distributed architecture is currently best for AI-native development

Rejected because: In this dataset, highly distributed architectures showed essentially no positive correlation between adoption and PR throughput.

Patterns

observability-tracing (measurement)

Jellyfish observes AI coding-tool impact across tool usage, PRs, source control, task systems, quality, and architecture rather than isolated model metrics.

eval-driven-development (evaluation)

The study evaluates AI coding-tool transformation through adoption, productivity, quality, and architecture-segmented metrics.

context-engineering (proposed)

The source uses context engineering to describe making cross-repository system relationships accessible to AI coding tools and agents, especially in distributed architectures.

human-in-the-loop-gate (workflow)

Interactive AI coding tools are treated as the production-ready path, with autonomous agents still mostly experimental rather than trusted for full-scale production.

Tradeoffs

How to measure adoption

Chose Developer AI adoption rate over Share of code generated by AI.

Rationale: Developer adoption captures behavior change and correlates more directly with productivity outcomes.

How to interpret PR throughput

Chose Track change within comparable contexts and segment by architecture over Compare absolute PR counts across organizations.

Rationale: Architecture affects natural PR volume, especially in highly distributed systems.

Which AI coding mode to prioritize today

Chose Interactive coding assistants over Full autonomous coding agents as the main production lever.

Rationale: Interactive tools show broad adoption and productivity gains, while autonomous agents remain early and low-volume in the data.

How to respond when productivity gains are missing

Chose Examine adoption and architecture/context constraints over Assuming the AI tools themselves failed universally.

Rationale: The data shows average gains, but highly distributed architectures underperform because tools lack cross-repo context.

Gotchas (what bit them)

High AI adoption does not guarantee productivity gains in fragmented codebases.

Root cause: Highly distributed architectures split relevant context across repositories and undocumented system relationships.

Resolution: Invest in context engineering and segment productivity analysis by architecture before drawing conclusions.

Lines-of-code metrics can misrepresent real AI adoption.

Root cause: Generated code volume does not directly capture whether developers actually use AI tools in their coding workflow.

Resolution: Measure developer adoption rate as the fraction of coding time that uses AI tools.

Autonomous agents lag far behind interactive coding tools in production usage.

Root cause: Most observed autonomous-agent usage is trialing and experimentation, not full-scale production.

Resolution: Set expectations around assistant-level tools first and track autonomous-agent adoption separately.

Productivity measurements can be confounded by architecture and work-scoping choices.

Root cause: More distributed architectures naturally require more PRs for migrations and cross-repo coordination.

Resolution: Measure changes within teams and segment by active repos per engineer instead of comparing absolute PR counts.

Quality effects are not absent forever just because early data is reassuring.

Root cause: The study had not found significant bug or revert increases yet, but asynchronous-agent usage is still evolving.

Resolution: Continue watching quality metrics as agent adoption grows.

Evaluation

The study evaluates AI transformation through weekly company-level snapshots across adoption rate, PR throughput, cycle time, PR size, bug tickets, PR reverts, bug-resolution rates, and active repositories per engineer. It then segments results by architecture to explain variance in productivity gains.

Metrics: AI-generated code share · developer AI adoption rate · company AI adoption rate · PR throughput per engineer per week · cycle time from first commit to merge · PR net lines of code added · files touched per PR · bug ticket creation · PR revert rate · bug resolution rate · active repos per engineer

Outcomes

• Median adoption rose from about 22% to close to 90%.
• Average PR throughput roughly doubled from 0% to 100% AI adoption.
• Cycle time decreased by 24% from 0% to 100% adoption.
• PRs became 18% larger by net lines of code added, without touching more files on average.
• No statistically significant relationship was found between AI adoption and bug creation or PR reverts.
• Bug-resolution rates increased because teams used AI for well-scoped bug backlog work.
• Centralized and balanced architectures saw about 4x PR-throughput gains, while highly distributed architectures showed essentially no positive correlation.

The study uses more than a year of telemetry from about 20 million PRs, 200,000 developers, and roughly 1,000 companies, with results spanning June 2024 to the presentation's present.