Autonomous PR Generation from Observability Data

posthog · Tech · 2026 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Product observability to code remediation

The team wanted to turn product signals, errors, logs, Slack messages, and session replays into actionable pull requests so engineers wake up to reviewed code changes instead of spending hours or days investigating dashboards and manually creating fixes.

Constraints:

• observability and product signals come from structurally different sources
• off-the-shelf embeddings group signals by source type rather than problem meaning
• some reports are too generic or product-sensitive for an agent to fix safely
• coding agents need repository context, CI feedback, PR comments, and rerun state
• generated PRs must remain reviewable, green, and tied to production outcomes

Architecture

The pipeline groups product signals into reports, runs a Claude Agent SDK research agent in a Modal sandbox with MCP tools, codebase context, Linear, and Notion context, then decides whether the issue needs human input or is immediately actionable. Actionable items are passed to a coding agent that clones the repo into a sandbox, writes the fix, pushes a PR, snapshots state, and rehydrates/reruns when CI or review comments arrive until the PR is green.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  product-signals[Product signals] --> query-generation-grouper[Query generation grouper]
  query-generation-grouper --> signal-report[Signal report]
  signal-report --> research-agent[Research agent]
  research-agent --> mcp-server[MCP server]
  research-agent --> codebase-context[Codebase context]
  research-agent --> external-mcps[External MCPs]
  research-agent --> actionability-step[Actionability step]
  actionability-step --> human-inbox[Human inbox]
  actionability-step --> coding-agent[Coding agent]
  codebase-context --> coding-agent
  coding-agent --> pull-request[Pull request]
  pull-request --> ci-and-review-feedback[CI and review feedback]
  ci-and-review-feedback --> sandbox-snapshot[Sandbox snapshot]
  sandbox-snapshot --> coding-agent

Components

Data flow

A product signal arrives from observability, errors, logs, Slack, or session replay. The system generates LLM queries from signals and matches those queries in embedding space to group related reports. Promoted reports go to a sandboxed research agent, which pulls extra context through MCP and produces a problem summary, priority, and reviewer. An actionability step either sends it to a human inbox or launches a sandboxed coding agent that opens and iterates on a PR until CI and comments are resolved.

Why this architecture

The pipeline separates signal grouping, research, actionability, and code execution because not every product signal is specific enough or safe enough for direct code changes. Query generation fixes cross-source clustering where embeddings grouped logs with logs and Slack with Slack. Sandboxed research and coding agents pull grounded context, while rehydrated snapshots close the CI/review loop so engineers see green PRs rather than intermediate failures.

Models used: Claude Agent SDK · LLM query generator · research agent · coding agent

Scale

• The manual process can take from a few hours to a few days.
  “From start to finish, this is going to take anywhere from a few hours to a few days, and it's not very interesting, but”transcript, stated
• The system can learn from repeated agent runs, even throwing the same problem at the agent 100 times during experimentation.
  “once you throw it at the same problem 100 times, you start seeing the kind of clever solutions that it comes up with”transcript, stated
• The product was in alpha with rollout planned over the next few months.
  “This is currently something that's in alpha. We'll be rolling it out kind of over the next few months. Um but where we really want to go is is”transcript, stated

Alternatives considered (roads not taken)

Manual dashboard-to-issue-to-PR workflow

Rejected because: It is slow, repetitive, and can take hours to days before a PR is reviewed and shipped.

Embed raw signals directly for clustering

Rejected because: Off-the-shelf embeddings grouped by structural source similarity rather than underlying product issue.

Send every signal report to a coding agent

Rejected because: Generic reports such as broad onboarding failures create noisy PRs that do not do meaningful work.

Avoid expensive agent calls early

Rejected because: During experimentation, repeated agent runs revealed patterns that could later be distilled into cheaper one-shot calls or faster models.

Patterns

query-rewriting (retrieval)

The system asks an LLM what a signal is about and embeds generated queries instead of raw source-shaped signals.

tool-use-orchestration (agentic)

Research and coding agents use MCP tools, codebase context, Linear, Notion, GitHub PRs, CI, and review comments.

human-in-the-loop-gate (decisioning)

Reports that require product judgment or are insufficiently actionable are routed to a human inbox for review.

durable-execution (agentic)

The coding sandbox is snapshotted and rehydrated when new CI failures or comments arrive.

eval-driven-development (quality)

CI failures and PR review comments trigger continued agent execution until the generated PR is green.

Tradeoffs

Use query-generation grouping

Chose LLM-generated semantic queries for embedding search over embedding raw observability signals directly.

Rationale: Generated queries reduced source-format clustering and grouped related issues across logs, Slack, and replays.

Route ambiguous reports to humans

Chose human inbox for product decisions or low-actionability reports over automatic PR generation for every signal cluster.

Rationale: Some reports are too generic or require product judgment that the agent should not make.

Rehydrate sandboxes on feedback

Chose snapshot and rerun the agent after CI or PR comments over manual local debugging of CI failures and comments.

Rationale: This closes the PR feedback loop so engineers wake up to green PRs rather than chores.

Gotchas (what bit them)

Raw embeddings grouped signals by format instead of issue.

Root cause: Errors, Slack messages, and session replays have structural similarity within source type that overwhelms semantic grouping.

Resolution: The team generated LLM queries from signals and matched those queries in embedding space.

Generic reports generate noisy PRs.

Root cause: If the problem is not specific enough, the coding agent will still try to fix something.

Resolution: An actionability step filters non-specific or product-decision reports into a human inbox.

Avoiding agent calls too early slowed learning.

Root cause: Cost concerns hid repeated behavior patterns that could later be optimized.

Resolution: The team ran agents during experimentation, then distilled expensive repeated steps into one-shot calls or faster models.

Evaluation

The source describes alpha-stage validation through grouped signal reports, research accuracy from MCP/codebase context, actionability filtering, PR greenness, CI/comment reruns, and production outcome learning after PR release.

Metrics: manual workflow takes a few hours to a few days · same-problem agent experimentation repeated 100 times · alpha rollout over the next few months · PRs iterated until green

Outcomes

• The pipeline groups product signals into reports and turns actionable reports into PRs.
• Research agents produce problem summaries, priority, and reviewer suggestions.
• Coding agents push PRs and rerun from snapshots on CI or review feedback.
• The product was in alpha and aimed toward self-improving product development loops.

The source discusses alpha maturity and workflow latency but does not report production PR counts or acceptance rates.