Actionable CI: Intelligent Analysis and Auto-Remediation of CI Pipeline Failures

block · Finance · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Developer experience

Block needed to make CI failures understandable and fixable for engineers and coding agents across complex repositories.

Constraints:

Large CI logs
Known and unknown failure modes
Autofix should apply only where eligible

Architecture

Block's Actionable CI is a three-layer CI intelligence system embedded in the developer workflow. Failed builds first pass through deterministic static-analysis rules for known patterns; remaining logs are bundled for an LLM that groups distinct issues and explains them with branch context; eligible failure classes can invoke headless Goose to generate a fix, open a draft PR, and validate it with CI retries. The CI results page exposes explanations, autofix, fix prompts, and raw logs to humans, while an MCP server and agent skills expose the same capabilities to coding agents.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ci-pipelines[CI pipelines] --> static-analysis-layer[Static analysis layer]
  static-analysis-layer --> llm-analysis-layer[LLM analysis layer]
  llm-analysis-layer --> cached-failure-analysis[Cached failure analysis]
  cached-failure-analysis --> ci-results-page[CI results page]
  ci-results-page --> autofix-eligibility-check[Autofix eligibility check]
  autofix-eligibility-check --> goose-autofix-agent[Goose autofix agent]
  goose-autofix-agent --> draft-pr[Draft PR]
  draft-pr --> draft-pr-validation[Draft PR validation]
  draft-pr-validation --> ci-pipelines
  ci-results-page --> mcp-server[MCP server]
  mcp-server --> agent-skills[Agent skills]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`CI pipelines`	Build and test systems whose failures produce overwhelming output.	`Static analysis layer`, `Draft PR validation`
`Static analysis layer`	Deterministic matcher for known failure patterns such as dependency conflicts and import violations.	`CI pipelines`, `LLM analysis layer`
`LLM analysis layer`	Groups failing-job logs into distinct issues and explains them in plain language with branch context.	`Static analysis layer`, `Cached failure analysis`
`Cached failure analysis`	Precomputed explanation available by the time developers open the CI result.	`LLM analysis layer`, `CI results page`, `Goose autofix agent`
`CI results page`	Developer-facing surface for Autofix, Explain, fix prompts, and raw logs.	`Cached failure analysis`, `Autofix eligibility check`, `MCP server`
`Autofix eligibility check`	Ensures every detected issue is supported and the issue count is under threshold before agent repair.	`CI results page`, `Goose autofix agent`
`Goose autofix agent`	Headless agent that generates fixes from the analysis.	`Cached failure analysis`, `Draft PR`
`Draft PR`	Pull request containing Goose's proposed changes.	`Goose autofix agent`, `Draft PR validation`
`Draft PR validation`	CI validation loop that passes, retries, or blocks promotion.	`Draft PR`, `CI pipelines`
`MCP server`	Programmatic interface exposing CI analysis and fix capabilities to coding agents.	`CI results page`, `Agent skills`
`Agent skills`	Agent-facing skills that let coding agents pull analysis, fix, and validate failures in a loop.	`MCP server`

Data flow

When CI fails, Actionable CI scans failed-job artifacts with static rules, sends unresolved logs to an LLM for grouped explanations, caches the analysis for the results page, checks whether all issues are supported and below threshold for autofix, invokes Goose to generate a fix, opens a draft PR, validates it with CI, retries on failure up to a limit, and promotes or closes the PR depending on CI and branch state.

3 supporting quotes

“When a CI build fails, the first step is entirely deterministic. We scan artifacts from failed jobs using static analysis rules that match common, well-known failure patterns.”article, stated

“After static analysis, we bundle the logs from failing jobs and send them to a language model with clear instructions to: 1. Identify the distinct issues present in the logs 2. Explain each issue in plain language, with awareness of the code changes on the branch”article, stated

“CI runs on the draft PR. If it passes, we move forward. If it fails, Goose analyzes the new failure and retries, up to a configured limit.”article, stated

Why this architecture

Block chose a layered design because CI failures have cheap known-knowns, ambiguous log clusters, and only some safe autofix categories. Static rules avoid unnecessary model cost, the LLM turns noisy logs into grouped explanations, Goose is only invoked after eligibility checks, and the results page/MCP split supports both humans and agents.

“We did not just throw a language model at build logs and call it a day. We built a pipeline with three distinct layers”article, stated

“This matters because language models are powerful, but they are slower and more expensive than a simple pattern match. If a failure can be identified with a regex, we do that.”article, stated

Models used: LLM analysis · Goose

Scale

The system targets thousands of engineers working across large interconnected repositories.
“"article" --- # Actionable CI At Block, we have thousands of engineers pushing code across large, interconnected repositories. Our CI pipelines are thorough, which means when something breaks, the failure output can be overwhelming.”article, stated
One broken import can create 15 test-suite failures, so grouping collapses noisy symptoms into one issue.
“A single broken import can cause 15 test suites to fail. Without grouping, a developer sees 15 problems. With grouping, they see one.”article, stated

Alternatives considered (roads not taken)

Send every failed build directly to an LLM

Rejected because: Known failures are faster and cheaper to classify with deterministic pattern matching.

“If a failure can be identified with a regex, we do that. Static analysis handles the easy cases and produces structured signal for the next layer.”article, stated

Autofix every CI failure

Rejected because: Only supported failure types under an issue-count threshold are safe enough for agentic repair.

“If any issue is not a good candidate, we skip autofix entirely. 2. Agent invocation The analysis from the first two layers is sent to a headless instance of Goose, with instructions to generate a fix. 3. Draft pull request Goose opens a draft PR with the proposed”article, stated

Keep the tool only developer-facing

Rejected because: Block wanted AI coding agents to iterate on CI failures programmatically through MCP and skills.

“This turns Actionable CI from a developer facing tool into infrastructure that both humans and agents can use.”article, stated

Patterns

`fallback-degradation` (analysis)

Cheap deterministic pattern matching handles known failures before the slower LLM path.

“This matters because language models are powerful, but they are slower and more expensive than a simple pattern match. If a failure can be identified with a regex, we do that. Static analysis handles the easy cases and produces structured signal for the next layer.”article, stated

`tool-use-orchestration` (agent-integration)

CI diagnosis and fix capabilities are exposed through MCP and skills for AI coding agents.

“We have built an MCP server and agent skills that expose CI analysis and fix capabilities programmatically, so AI coding agents can iterate on CI failures without a developer needing to manually check the results.”article, stated

Tradeoffs

When to use an LLM in CI analysis.

Chose static analysis before LLM analysis over sending every failure straight to a model.

Rationale: Known failures are faster and cheaper to classify deterministically.

How much CI repair to automate.

Chose explicit developer-triggered autofix for eligible failures over automatic repair attempts on every failure.

Rationale: Block limits agentic repair to supported categories and keeps developers in control from the CI page.

“Autofix is initiated explicitly from the CI results by the developer.”article, stated

How to serve both humans and coding agents.

Chose CI page UX plus MCP/skills API over a human-only results page.

Rationale: The same analysis and fix loop becomes infrastructure for agents when exposed programmatically.

“An agent can pull the failure analysis, generate a fix, and validate it against CI, all in a loop.”article, stated

Gotchas (what bit them)

Autofix could not safely cover every CI failure category.

Root cause: Only certain failures had enough structure and validation support to let an agent open and retry a draft fix.

Resolution: Block exposed explanation, fix-prompt, and raw-log paths for failures that were not eligible for autofix.

“For failures that are not eligible for autofix, the Results page gives developers the tools to move fast. Each issue has an Explain button that generates a plain-language breakdown”article, stated

Evaluation

The pipeline grounds explanations in code changes and logs, with autofix eligibility boundaries.

Metrics: known-pattern classification · LLM explanations · eligible autofixes

“Each issue has an Explain button that generates a plain-language breakdown grounded in the actual code changes and logs.”article, stated

Outcomes

Developers and agents can diagnose and remediate CI failures from the CI results workflow.

The system is integrated into CI pages and programmatic agent access.

“We built this directly into our CI results page, so it feels like part of the workflow, not a separate tool.”article, stated