2x Engineering Throughput Through AI-First Development Platform

intercom · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Engineering productivity

Intercom needed to increase R&D throughput while remaking its product and engineering culture around AI-native development.

Constraints:

Architecture

Intercom turned Claude Code adoption into an internal agent-development platform. Engineers work in Claude Code, but skills, hooks, and a create-PR workflow inject team standards before code reaches review; Honeycomb telemetry and uploaded Claude Code session JSON provide product-style observability over skill use and failure modes; LLM judges evaluate PR-description quality; and existing CI/CD and GitHub checks remain downstream gates. The system is deliberately platform-shaped: small reusable skills and telemetry loops make agent-first work more deterministic across the R&D organization.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  claude-code[Claude Code] --> internal-skills-repository[Internal skills repository]
  internal-skills-repository --> hooks[Hooks]
  claude-code --> create-pr-skill[Create PR skill]
  create-pr-skill --> github-pr-workflow[GitHub PR workflow]
  github-pr-workflow --> ci-cd-pipeline[CI/CD pipeline]
  internal-skills-repository --> honeycomb-telemetry[Honeycomb telemetry]
  claude-code --> claude-code-session-json[Claude Code session JSON]
  claude-code-session-json --> s3-session-lake[S3 session lake]
  s3-session-lake --> agent-use-analysis[Agent-use analysis]
  create-pr-skill --> llm-pr-judge[LLM PR judge]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Claude Code Primary coding agent used by Intercom engineers for agent-first development. Internal skills repository, Create PR skill, Claude Code session JSON
Internal skills repository Collection of reusable instructions and scripts that standardize agent behavior across workflows. Claude Code, Hooks, Honeycomb telemetry
Hooks Lightweight enforcement and guidance points that steer agent workflows toward approved practices. Internal skills repository, Claude Code
Create PR skill Opinionated workflow skill for generating PR descriptions that explain intent and review context. Claude Code, GitHub PR workflow, LLM PR judge
GitHub PR workflow Downstream pull-request workflow where generated code and descriptions are reviewed and checked. Create PR skill, CI/CD pipeline
CI/CD pipeline Existing deterministic checks, lints, preflight checks, and deployment gates after code is written. GitHub PR workflow
Honeycomb telemetry Event-level dashboard for skill invocation and adoption metrics. Internal skills repository
Claude Code session JSON Raw local session records uploaded for anonymized organizational analysis. Claude Code, S3 session lake
S3 session lake Storage layer for uploaded Claude Code session data used to inspect recurring agent issues. Claude Code session JSON, Agent-use analysis
Agent-use analysis Internal evaluation of how engineers use Claude Code and where team-level fixes are needed. S3 session lake, Internal skills repository
LLM PR judge Judge model that scores pull-request description quality over historical data. Create PR skill

Data flow

An engineer works in Claude Code, which invokes internal skills and scripts at workflow steps such as pull-request creation. The create-PR skill uses available context to produce higher-quality PR descriptions, GitHub/CI checks continue to gate the output, skill invocations are sent to Honeycomb, and raw Claude Code session JSON is uploaded to S3 for anonymized, organization-level analysis of adoption and recurring agent failures.

3 supporting quotes
“a skill is basically just like a set of instructions and sometimes scripts that a LLM or a agent harness can invoke at a certain step in your flow.”transcript, stated
“So that individuals one can triage their own implementation as you said oh it looks like I need to do this or that or improve my agents MD.”transcript, stated
“you all are uploading those files to S3 and then layering on top of it some anonymization some user level views”transcript, stated

Why this architecture

Intercom chose a platform of skills, hooks, telemetry, and judges because raw agent use alone did not guarantee team standards. Skills move deterministic guardrails upstream into the coding process, telemetry makes adoption and failure modes visible, and LLM judges keep quality from sliding as Claude Code produces most code.

3 supporting quotes
“What I think is really interesting about skills is you can bring some of that determinism to as you write the code how you want that process to go.”transcript, stated
“treating your org like a product treating your repo like a product treating your AI setup as a team like a product”transcript, implied
“One of the things we noticed early on when we started getting Cloud Code to write all of our code, and you know, we're well above 90% now, is that it would create pull request descriptions that were kind of terrible.”transcript, stated

Models used: Claude Code · LLM judges

Scale

Alternatives considered (roads not taken)

Document standards in wikis and SOPs

Rejected because: Written process docs depended on humans remembering and applying them, whereas skills can make the golden path easier to enforce inside the agent workflow.

“we would do this by writing wikis or having, you know, SOPs where it said, "Can you please follow step A B C D E?"”transcript, stated

Let Claude Code write generic PR descriptions

Rejected because: The generic descriptions summarized code rather than intent, which was not useful enough for human or agent reviewers.

“it would create pull request descriptions that were kind of terrible. They it would describe the code, and that's the least interesting part of a pull request.”transcript, stated

Optimize token spend early

Rejected because: Intercom accepted high model spend during the adoption phase because the perceived throughput alpha was worth learning faster.

“we think that there's enough alpha or benefit in at this point going as fast as possible and caring about the bill later”transcript, stated

Patterns

tool-use-orchestration (agent-platform)

Internal skills and hooks steer Claude Code toward approved engineering workflows such as pull-request creation.

“we started we told people like, "Oh, just use the create PR skill." And then people would want to use it. You don't”transcript, stated

observability-tracing (operations)

Skill and session telemetry are treated as product instrumentation for adoption and quality management.

“everybody every day. Um find a way to put event level to telemetry invoked in the skill a little dashboard”transcript, stated

Tradeoffs

How aggressively to optimize AI cost during adoption.

Chose speed and learning over early token-cost optimization.

Rationale: The source frames high model spend as an investment while throughput was increasing.

“However, their attitude has been to treat this as an investment and prioritize speed over optimization at this stage.”summary, stated

How to enforce engineering standards in agent-authored work.

Chose skills and hooks embedded in Claude Code workflows over wiki/SOP guidance that humans must remember.

Rationale: Intercom treated skills as upstream guardrails analogous to CI/CD, making standards easier to follow before code reached review.

“now you can just make it really easy to enforce those standards across a team”transcript, stated

How to improve PR descriptions from agent-generated code.

Chose a create-PR skill plus LLM judging over accepting Claude Code's default code-summary descriptions.

Rationale: Default descriptions lost the intent reviewers needed, so Intercom codified its PR-description expectations and measured quality over historical data.

“you want to know the intent behind the pull request. You want to know the interesting bits, what's kind of related to this”transcript, stated

Gotchas (what bit them)

Code review became the next bottleneck after AI increased output.

Root cause: Throughput improvements shifted pressure to downstream human review capacity.

Resolution: Intercom kept quality checks and CI in the path while identifying new bottlenecks.

“After fixing bottlenecks and improving performance, CI is no longer the bottleneck, though code review has become one.”summary, stated

Evaluation

Intercom used PR throughput, LLM judges, telemetry, and external code-quality analysis.

Metrics: PRs per R&D head · LLM judge scores · skill usage

“The PR description quality judge is one example.”summary, stated

Outcomes

The platform was rolled out across the R&D organization with hundreds of skills and telemetry.

“The headline result is doubling merge PRs per R&D head over nine months, measured from nine months before spring 2026 to spring 2026.”summary, stated