AI-Powered On-Call Assistant for Airflow Pipeline Debugging

wix · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Data engineering on-call and Airflow operations

Wix needed to reduce the operational load of debugging Airflow DAG failures across a large, heterogeneous data platform where generic alerts forced engineers to manually connect Airflow, Spark, Kubernetes, code, schema, and ownership context.

Constraints:

• the platform serves 250 million users and generates 4 billion HTTP transactions daily
• Wix operates a 7 petabyte data lake and more than 3,500 Airflow DAGs
• even 99.9% reliability creates daily failures at this scale
• internal tools must avoid exposing public inbound endpoints
• SRE automation needs structured, parseable outputs rather than free text
• different tasks require different model cost and reasoning tradeoffs

Architecture

Wix built AirBot as an AI-powered Slack on-call agent. Slack Socket Mode keeps the bot inside the private network through an outbound WebSocket, while Slack Bolt Python and FastAPI manage connectivity. The bot gathers operational context through MCP-style integrations with GitHub, Trino, Spark, OpenMetadata, DDS lineage, and ownership tags. A LangChain Chain-of-Thought flow classifies the operator and error category, analyzes code and logs, and generates a remediation plan or pull request. GPT-4o Mini handles high-volume classification, Claude 4.5 Opus handles complex root-cause analysis and PR generation, and Pydantic structured outputs make downstream automation reliable. Deployment uses Docker, serverless infrastructure, and Vault-managed secrets.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  opsgenie-airflow-alerts[opsgenie-airflow-alerts] --> slack-airbot[slack-airbot]
  slack-airbot --> socket-mode-connection[socket-mode-connection]
  socket-mode-connection --> private-wix-network[private-wix-network]
  private-wix-network --> mcp-context-tools[mcp-context-tools]
  private-wix-network --> vault-secrets[vault-secrets]
  slack-airbot --> reasoning-chains[reasoning-chains]
  mcp-context-tools --> reasoning-chains
  reasoning-chains --> model-router[model-router]
  reasoning-chains --> structured-output-model[structured-output-model]
  structured-output-model --> github-pr-workflow[github-pr-workflow]
  github-pr-workflow --> slack-airbot

Components

Data flow

An Airflow/Opsgenie alert reaches AirBot in Slack over Socket Mode. AirBot gathers task, code, log, schema, lineage, ownership, and metric context from internal systems. The classification chain identifies operator and error type, the analysis chain determines root cause from logs and code, and the solution chain emits a structured remediation plan or candidate PR. For schema mismatches it can fetch SQL from GitHub and schema from OpenMetadata, open a PR, and present a review button in Slack; for upstream delays it can query internal APIs and identify the blocking upstream table.

Why this architecture

Hard-coded alert automation failed because Airflow orchestrates multiple execution systems and internal services. Wix therefore used an agentic Slack workflow that can collect context from heterogeneous tools, reason over logs and code, and produce structured remediation. Socket Mode was chosen to protect the network boundary, tiered model routing balances cost against reasoning depth, and Pydantic narrows nondeterministic LLM output into a reliable automation contract.

Models used: GPT-4o Mini · Claude 4.5 Opus · LangChain · Pydantic

Scale

• Wix serves 250 million users.
  “Wix operates a massive cloud-based development platform serving 250 million users.”article, stated
• The platform generates more than 4 billion HTTP transactions daily.
  “This generates over 4 billion HTTP transactions daily, feeding a data lake that currently holds 7 petabytes of data.”article, stated
• Wix maintains over 3,500 Apache Airflow DAGs.
  “To manage this volume, Wix Data Engineering maintains over 3,500 Apache Airflow pipelines (DAGs).”article, stated
• The reported operational measurement covers 30 team channels and 60 data engineers.
  “metrics gathered from 30 team channels supporting 60 Data Engineers show a measurable shift in our operational efficiency”article, stated
• AirBot generated 180 candidate pull requests, with 28 merged directly and a 15% fully automated fix rate.
  “AirBot generated 180 candidate Pull Requests across our main repositories. While 28 were merged directly without human code changes”article, stated
• A typical debugging cycle was about 45 minutes and AirBot cut at least 15 minutes per incident.
  “A typical manual debugging cycle takes ~45 minutes. AirBot cuts this by at least 15 minutes per incident.”article, stated
• The reported savings were 675 engineering hours per month, equivalent to about 4 full-time engineers.
  “* Total Savings: 2,700 × 0.25 hours = 675 engineering hours saved per month.”article, stated
• Average AirBot AI interaction workflow cost was about $0.30.
  “An average AirBot AI interaction workflow costs ~$0.30.”article, stated

Alternatives considered (roads not taken)

traditional Slack HTTP request endpoints

Rejected because: Public inbound endpoints would require opening the security perimeter for Slack events.

hard-coded tool-specific integrations

Rejected because: The Airflow stack spans Trino, Spark, Kubernetes, and internal services, making brittle integrations hard to scale.

passive bot that reposts error logs

Rejected because: Wix wanted active investigations that classify, analyze, and propose fixes rather than static notifications.

free-text LLM remediation output

Rejected because: Automation needs parseable, typed responses that downstream code can reliably use.

4.4 Deployment & Infrastructure

To ensure reliability and maintainability, the application infrastructure is built on modern DevOps principles:"

— article, stated

Patterns

tool-use-orchestration (agent-architecture)

AirBot gathers context and acts across GitHub, Trino, Spark, OpenMetadata, DDS lineage, ownership tags, Slack, and PR workflows.

prompt-chaining (agent-architecture)

The reasoning engine separates alert classification, root-cause analysis, and remediation generation into chained steps.

model-routing (model-operations)

AirBot uses cheaper GPT-4o Mini for high-volume classification and Claude 4.5 Opus for complex root-cause reasoning and PR generation.

structured-output-contract (integration)

Pydantic models force model responses into typed remediation objects for downstream automation.

human-in-the-loop-gate (operations)

Even when AirBot creates a PR, engineers review candidate fixes through Slack/GitHub before merging unless fully automated fixes are accepted.

"It identifies the mismatch.

• Resolution: It opens a PR swapping the incorrect column for the correct one (r.start_date) and presents a "Review PR" button in Slack."

cost-aware-routing (model-operations)

The model-selection split and reported $0.30 average workflow cost keep operational automation economically attractive.

Tradeoffs

Use Slack Socket Mode for connectivity.

Chose outbound WebSocket from inside the private network over public inbound Slack HTTP endpoint.

Rationale: Socket Mode preserves interaction with Slack without opening inbound firewall ports.

Split reasoning by chain stage.

Chose classification, analysis, and solution chains over one monolithic model call.

Rationale: The workflow needs to identify the failing operator, inspect code/log context, and generate a remediation plan as separate reasoning tasks.

Route models by task complexity.

Chose GPT-4o Mini for classification and Claude 4.5 Opus for deep reasoning over using the same expensive model for every step.

Rationale: This keeps common classification fast and cheap while reserving stronger reasoning for larger-context root-cause and PR-generation tasks.

Generate PRs as candidate blueprints.

Chose candidate pull requests with human review affordances over always-auto-merge remediation.

Rationale: Only a subset of fixes merged directly; many PRs still helped engineers reason faster even when they chose manual implementation.

Gotchas (what bit them)

Traditional alerting leaves engineers as cross-system error parsers.

Root cause: Airflow failures require connecting generic alerts, UIs, distributed logs, code changes, and ownership context.

Resolution: Have the agent collect context from internal tools and synthesize root cause before handing engineers a fix path.

Internal chat agents can accidentally weaken the network perimeter.

Root cause: Standard Slack event delivery sends public HTTP requests to an exposed URL.

Resolution: Use Socket Mode so the bot initiates outbound WebSocket connectivity from inside the private network.

LLM free text is too unreliable for SRE remediation automation.

Root cause: LLMs are nondeterministic, while downstream systems need predictable fields.

Resolution: Use Pydantic output models and a remediation-plan schema.

One model cannot economically handle every SRE subtask.

Root cause: Log classification is high-volume and simple, while root-cause analysis and PR generation require deeper reasoning and larger context.

Resolution: Route high-volume classification to GPT-4o Mini and complex analysis to Claude 4.5 Opus.

Evaluation

Wix evaluates AirBot with operational metrics over team-channel usage, pull-request generation and merge rates, incident time saved, user feedback, monthly engineering-hours saved, and per-interaction cost.

Metrics: 180 candidate pull requests · 28 direct merges · 15% fully automated fix rate · 4,200 successful flows · 66% positive feedback rate · 675 engineering hours saved per month · $0.30 average workflow cost

Outcomes

• AirBot generated 180 candidate PRs in the measured window.
• Twenty-eight PRs merged directly without human code changes.
• The system saved an estimated 675 engineering hours per month.
• The average AI workflow cost was about $0.30.
• The assistant shifted on-call work from manual log parsing toward proactive decision support.

The source provides unusually concrete production scale: 250 million users, 4 billion daily HTTP transactions, a 7PB data lake, 3,500 DAGs, 30 team channels, 60 data engineers, 4,200 successful flows, and 675 monthly hours saved.