AI-Powered Root Cause Analysis Assistant for Race Day Operations

formula_1 · Automotive · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Race-day IT operations and incident response

Formula 1 needed to shorten root-cause analysis for critical live-event IT issues that crossed APIs, networks, databases, logs, incident tools, and multiple engineering teams.

Constraints:

• race events created high-stakes operational pressure
• critical issues crossed development, operations, infrastructure, and networking teams
• change freezes and event schedules extended fixes over weeks
• engineers needed natural-language access without deep domain-specific system knowledge
• agentic database and infrastructure checks had to be controlled to avoid hallucinated or risky actions

Architecture

F1 and AWS built a chat-based RCA assistant on Amazon Bedrock. Raw logs flow into S3, hourly EventBridge jobs trigger AWS Glue and Spark ETL, transformed logs are stored in S3 and ingested into Amazon Bedrock Knowledge Bases, and Amazon Bedrock Agents use Claude 3 Sonnet to plan, retrieve, call controlled tools, and generate troubleshooting answers. The agent can query knowledge bases, approved SQL checks, Boto3 API checks, Datadog, and Jira, while a Streamlit UI exposes the final answer and a hidden orchestration trace for review.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  raw-log-s3-bucket["raw-log-s3-bucket"] --> eventbridge-etl-scheduler["eventbridge-etl-scheduler"]
  eventbridge-etl-scheduler --> glue-spark-etl["glue-spark-etl"]
  glue-spark-etl --> transformed-log-s3-bucket["transformed-log-s3-bucket"]
  transformed-log-s3-bucket --> bedrock-knowledge-bases["bedrock-knowledge-bases"]
  bedrock-knowledge-bases --> bedrock-rca-agent["bedrock-rca-agent"]
  bedrock-rca-agent --> controlled-health-check-tools["controlled-health-check-tools"]
  bedrock-rca-agent --> streamlit-chat-ui["streamlit-chat-ui"]

Components

Data flow

Raw operational logs are centralized in S3. EventBridge checks hourly for new files and starts Glue/Spark ETL to standardize schemas, remove unused columns, and aggregate data at minute granularity. Transformed logs are stored in another S3 bucket and fed into Bedrock Knowledge Bases. A user asks the Streamlit chat assistant a natural-language question; Bedrock Agents create an execution plan, retrieve relevant documents, invoke controlled database/API/monitoring/incident tools, consolidate tool and retrieval outputs, and generate a final answer with a trace available for inspection.

Why this architecture

The architecture mirrors F1's manual troubleshooting process while making it queryable through a managed agentic RAG stack. ETL reduces noisy log volume before retrieval, Knowledge Bases grounds answers in transformed logs, Bedrock Agents add multi-step planning and tool invocation, and controlled SQL/API checks preserve least-privilege safety. The Streamlit UI separates concise answers from traces so engineers can inspect retrieval and tool execution when needed.

Models used: Amazon Bedrock Agents · Amazon Bedrock Knowledge Bases · Anthropic Claude 3 Sonnet · AWS Glue · Apache Spark · Streamlit

Scale

• Critical issue triage, testing, and resolution could take up to 3 weeks before the assistant.
  “it can take up to 3 weeks to triage, test, and resolve a critical issue”article, stated
• One recurring web API issue consumed around 15 full engineer days over several events.
  “Teams spent around 15 full engineer days to iteratively resolve the issue over several events”article, stated
• The prototype phase lasted 5 weeks and covered two candidate systems.
  “F1 embarked on a 5-week prototype to demonstrate the feasibility of this solution.”article, stated

  “replicate and automate the current manual troubleshooting process for two candidate systems.”article, stated
• The RCA tool reduced one issue to 3 days including deployments and testing over a race weekend.
  “the team could narrow down the root cause and implement a solution within 3 days, including deployments and testing over a race weekend.”article, stated
• Query responses arrive within 5-10 seconds, with initial triage reduced from more than a day to less than 20 minutes.
  “engineers can receive a response within 5–10 seconds on a specific query and reduce the initial triage time from more than a day to less than 20 minutes.”article, stated
• End-to-end time to resolution was reduced by as much as 86%.
  “The end-to-end time to resolution has been reduced by as much as 86%.”article, stated
• Log aggregation reduced 60 response-time points per minute to one per-minute event.
  “when there are 60 data points per minute with API response times, data was aggregated to a single data point per minute.”article, stated

Alternatives considered (roads not taken)

manual cross-team incident investigation

Rejected because: It required weeks of triage, log review, anomaly inspection, and iteration across several teams and live events.

feeding raw logs directly to the assistant

Rejected because: The system needed standardized formats, less irrelevant data, and lower retrieval cost before knowledge-base ingestion.

letting the LLM create ad hoc SQL queries or health checks

Rejected because: That would increase hallucination, prompt-injection, and risky-syntax exposure; F1 used predefined least-privilege checks instead.

Patterns

rag-retrieval (knowledge-grounding)

Transformed logs are ingested into Bedrock Knowledge Bases so the RCA assistant can retrieve operational context before answering.

tool-use-orchestration (agent-architecture)

Bedrock Agents create troubleshooting plans and invoke databases, EC2 checks, Datadog, Jira, and knowledge bases as tools.

guardrails-io-filtering (safety)

The system constrains agent actions to controlled SQL/API checks, schema validation, encryption, and least-privilege access.

observability-tracing (operations)

The chat UI exposes hidden orchestration traces so engineers can inspect the tools and documents behind an answer.

Tradeoffs

Aggregate logs to minute-level granularity.

Chose one aggregated event per minute over keeping all per-request response-time points in retrieval.

Rationale: Minute-level aggregation preserved what users needed to find incident timing while reducing the amount of data the assistant had to analyze.

Use controlled tool APIs rather than model-generated queries.

Chose predefined SQL and API checks over LLM-authored database queries and system health checks.

Rationale: The controlled-tool design reduced hallucination and prompt-injection risk while still letting the assistant inspect live system health.

Hide orchestration traces by default while keeping them reviewable.

Chose clean final answer plus optional trace inspection over always exposing the full agent execution trace.

Rationale: The UI keeps responses concise for routine use but lets engineers inspect tool and document choices when correctness matters.

Gotchas (what bit them)

Raw logs were not directly suitable for cost-effective, accurate RCA retrieval.

Root cause: Different sources used varied structures, schemas, column names, timestamp formats, and unnecessary fields.

Resolution: Standardize, filter, and aggregate logs before knowledge-base ingestion.

Agent-created health checks would be unsafe for production systems.

Root cause: Hallucinations, prompt injections, and risky syntax could affect databases or infrastructure queries.

Resolution: Implement one explicit function per health check with least-privilege access and schema controls.

A concise chat answer could hide whether the agent used the right tools and documents.

Root cause: Agentic RAG involves multiple retrieval and tool steps whose correctness is not visible in a final answer alone.

Resolution: Show a hidden trace panel for orchestration plans, executions, invoked tools, and retrieved documents.

Evaluation

F1 and AWS first ran a 5-week prototype over two candidate systems, then measured operational impact through response latency, initial triage time, end-to-end resolution time, and a recurring web API issue.

Metrics: 5-week prototype · two candidate systems · 5-10 second query response · initial triage from more than a day to less than 20 minutes · end-to-end time to resolution reduced by as much as 86% · recurring issue reduced from 15 engineer days to 3 days including deployment/testing

Outcomes

• engineers could query operational issues in natural language
• initial triage time dropped from more than a day to less than 20 minutes
• end-to-end resolution time was reported as reduced by up to 86%
• difficult issues could be routed or escalated to the appropriate F1 engineering team

The strongest reported numbers are 5-10 second query responses, triage below 20 minutes, 3-day resolution for a previously 15-engineer-day issue, and up to 86% end-to-end reduction.