LLM-Powered Security Incident Response and Automation

agoda · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Security incident response operations

Agoda needed to reduce repetitive security-operations workload across alert triage, phishing reports, and incident documentation while preserving conservative human oversight for high-stakes incident response.

Constraints:

Architecture

Agoda built three targeted LLM-powered security workflows: a RAG-backed alert triage system that retrieves similar incidents and produces summaries, impact assessments, and escalation verdicts; a fully autonomous phishing classifier that enriches reports with header and metadata intelligence before responding in real time; and a human-reviewed incident-report generator that consolidates investigation context into structured reports. The systems are integrated into existing security pipelines and tuned toward conservative escalation.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Alert["Security alert pipeline"] --> Context["Alert context collector"]
  Context --> Retriever["Historical incident retriever"]
  Retriever --> Triage["Alert triage LLM"]
  Context --> Triage
  Triage --> Escalation["Human escalation review"]
  Parser["Phishing enrichment parser"] --> Phish["Phishing classifier LLM"]
  Phish --> Response["Automated response sender"]
  Incident["Incident context collector"] --> Report["Incident report LLM"]
  Report --> Reviewer["Human report reviewer"]
Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component Role Interacts with
Security alert pipeline Receives recurring security alerts and triggers triage automation from Agoda's existing monitoring flow. Alert context collector
Alert context collector Gathers event timelines and metadata needed to analyze an alert. Security alert pipeline, Historical incident retriever, Alert triage LLM
Historical incident retriever Queries a vector database of past incidents and root-cause analyses for relevant cases. Alert context collector, Alert triage LLM
Alert triage LLM Generates security-alert summaries, impact assessments, and escalation verdicts. Alert context collector, Historical incident retriever, Human escalation review
Phishing enrichment parser Parses reported emails and enriches them with header, metadata, and threat-intelligence signals. Phishing classifier LLM
Phishing classifier LLM Classifies submitted emails as phishing, spam, or safe and triggers real-time reporter responses. Phishing enrichment parser, Automated response sender
Incident context collector Collects investigation context from communication and work-tracking systems after an incident is resolved. Incident report LLM
Incident report LLM Drafts structured incident timelines, detection signals, impacts, and resolution steps. Incident context collector, Human report reviewer
Human escalation review Reviews over-escalated or high-stakes cases to avoid missed threats. Alert triage LLM, Incident report LLM
Human report reviewer Validates drafted incident reports before publication to internal documentation. Incident report LLM
Automated response sender Sends real-time classification responses to phishing reporters. Phishing classifier LLM

Data flow

Security alerts enter Agoda's existing pipeline, have timelines and metadata collected, retrieve similar historical cases from a vector database of past incidents and root-cause analyses, and are summarized by an LLM into impact and escalation judgments. Phishing reports are parsed, enriched with threat intelligence, classified by an LLM, and answered automatically. Resolved incidents gather context from investigation systems and communication channels, then an LLM drafts structured reports for human validation.

"Trigger: Alerts enter the system through Agoda’s existing security pipeline.

3 supporting quotes
“Historical Reference: A Retrieval-Augmented Generation (RAG) system queries a vector database of past incidents and root cause analyses.”article, stated
“LLM Inference: The model receives the alert timeline, metadata, and similar past cases. It’s prompted to emulate a seasoned security analyst.”article, stated
“When a phishing report comes in, the email is parsed and enriched with threat intelligence based on its headers and metadata.”article, stated

— article, stated

Why this architecture

Agoda avoided a single broad security automation system and instead built narrow workflows around repeatable, measurable operational bottlenecks. RAG gives alert triage organization-specific historical context, autonomous phishing classification is used where the process is structured and monitored, and report generation retains human review because post-incident documentation must be accurate and compliant.

4 supporting quotes
“Rather than automate everything at once, we focused on targeted, high-impact workflows that had clear structure and repeatable patterns.”article, stated
“Each was built around a specific problem, designed to be easy to measure, and integrated directly into our existing workflows.”article, stated
“Alerts are automatically enriched with historical context and analyzed using a retrieval-augmented LLM workflow”article, stated
“Still, human oversight is crucial, especially when high-stakes decisions are involved.”article, stated

Models used: LLMs · retrieval-augmented LLM workflow

Scale

Alternatives considered (roads not taken)

Comprehensive automation of all security operations

Rejected because: The team chose targeted high-impact workflows with clear structure and measurable outcomes instead.

“Rather than automate everything at once, we focused on targeted, high-impact workflows that had clear structure and repeatable patterns.”article, stated

Manual analyst triage for recurring Level 1-2 alerts

Rejected because: Manual triage took 20 to 40 minutes per alert and consumed analyst capacity on repetitive investigations.

“Prior to automation, this process involved manually reviewing logs, correlating historical incidents, and writing investigation summaries, each alert taking 20 to 40 minutes to handle.”article, stated

Manual review of every user-submitted phishing report

Rejected because: Most reports were harmless, yet every report still required analyst time and delayed user feedback.

“Before automation, each email was reviewed manually by an analyst to determine whether it was malicious, benign, or simply spam.”article, stated
“While most submissions were safe, each report required time and attention from an analyst, creating operational overhead and delaying feedback to users.”article, stated

Rely solely on the model when context is incomplete

Rejected because: For high-stakes security, Agoda routes incomplete-context cases to human review and favors over-escalation.

“When the LLM doesn’t have access to full context — such as prior alerts or historical signals — it’s better to over-escalate and route the case for human review”article, stated

Patterns

rag-retrieval (retrieval)

The alert triage workflow retrieves similar past incidents and root-cause analyses to ground LLM analysis in Agoda-specific security history.

“Historical Reference: A Retrieval-Augmented Generation (RAG) system queries a vector database of past incidents and root cause analyses.”article, stated
“Alerts are automatically enriched with historical context and analyzed using a retrieval-augmented LLM workflow”article, stated

human-in-the-loop-gate (safety)

Agoda keeps humans in high-stakes decision paths: cases with incomplete context are over-escalated, and incident reports are validated before publication.

“Still, human oversight is crucial, especially when high-stakes decisions are involved.”article, stated
“When the LLM doesn’t have access to full context — such as prior alerts or historical signals — it’s better to over-escalate and route the case for human review”article, stated

tool-use-orchestration (orchestration)

The systems connect LLM calls to security pipelines, vector databases, email parsing/enrichment, threat intelligence, documentation, and response-sending workflows.

“The manual review process involved reading the email body, analyzing headers, cross-checking against threat intelligence feeds, and then responding to the user with a verdict.”article, stated
“Gathering the necessary context often meant reviewing Slack conversations, Jira tickets, Confluence pages, Teams meeting transcripts, and email chains”article, stated

eval-driven-development (evaluation)

The workflows are measured through human-alignment, phishing precision/false-negative, factual-accuracy, speed, and workload metrics.

“Internal validation over the last three months shows a 97%+ alignment with human analyst conclusions during peer review.”article, stated
“In production, it has maintained over 99% classification precision, correctly identifying both malicious and benign submissions.”article, stated

safety-policy-enforcement (safety)

The alert triage system is deliberately conservative, minimizing missed threats by favoring over-escalation for human review.

“The system is tuned to favor over-escalation, minimizing the risk of missed threats.”article, stated
“In practice, false positives are quickly reviewed and closed, and no high-impact incidents have been misclassified since rollout.”article, stated

Tradeoffs

Automate targeted security workflows instead of all incident response

Chose three focused systems for alert triage, phishing classification, and incident reports over attempting comprehensive security-operations automation.

Rationale: The narrower approach matched high-volume repeatable pain points with measurable outcomes and workflow integration.

“Rather than automate everything at once, we focused on targeted, high-impact workflows that had clear structure and repeatable patterns.”article, stated
“Each was built around a specific problem, designed to be easy to measure, and integrated directly into our existing workflows.”article, stated

Favor over-escalation for alert triage

Chose conservative escalation and human review when context is incomplete over maximizing automation by trusting limited-context model judgments.

Rationale: False positives are cheaper to close than missed high-impact security incidents.

“The system is tuned to favor over-escalation, minimizing the risk of missed threats. In practice, false positives are quickly reviewed and closed”article, stated
“When the LLM doesn’t have access to full context — such as prior alerts or historical signals — it’s better to over-escalate and route the case for human review”article, stated

Fully automate phishing responses but review incident reports

Chose 24/7 autonomous phishing classification and human-reviewed report generation over using the same automation level for every security workflow.

Rationale: Phishing classification had strong precision and immediate-response benefits, while incident reports remain compliance and knowledge-sharing artifacts that need validation.

“This system now runs fully autonomously, 24/7. It classifies emails and responds in under 25 seconds”article, stated
“A reviewer validates the content and publishes the final version to our internal documentation system.”article, stated

Gotchas (what bit them)

Security LLMs can make conflicting or variable judgments.

Root cause: LLMs are non-deterministic, so the same input can produce different or conflicting outputs.

Resolution: Use cross-checking where helpful and keep human oversight for high-stakes decisions.

“Large Language Models (LLMs) are non-deterministic by nature — meaning the same input can sometimes produce different or even conflicting outputs.”article, stated
“To increase confidence in results, it’s often helpful to have one LLM cross-check the response of another. Still, human oversight is crucial”article, stated

Incomplete context can turn automation into unsafe under-escalation.

Root cause: The LLM may lack prior alerts or historical signals needed to judge a security event.

Resolution: Favor over-escalation and route incomplete-context cases to human review.

“When the LLM doesn’t have access to full context — such as prior alerts or historical signals — it’s better to over-escalate and route the case for human review rather than rely solely on the model’s limited understanding.”article, stated

Dumping too much data into the model can degrade incident analysis.

Root cause: LLMs can lose context and make mistakes when supplied with excessive data.

Resolution: Provide accurate, concise, well-documented, structured, and enriched input instead of exhaustive data dumps.

“While models don’t require exhaustive data dumps, the more structured and enriched the input, the stronger and more reliable the analysis.”article, stated
“They can help find a needle in the haystack in case of large data, but they are also capable of losing context and getting it wrong when supplied with too much data.”article, stated

Incident documentation quality can vary when analysts must manually consolidate scattered sources.

Root cause: Security context is distributed across communication, ticketing, meeting, and email systems.

Resolution: Use an LLM workflow to collect and summarize the timeline, detection signals, impact, and resolution steps, then have a reviewer validate the report.

“Gathering the necessary context often meant reviewing Slack conversations, Jira tickets, Confluence pages, Teams meeting transcripts, and email chains, sometimes across multiple teams.”article, stated
“A reviewer validates the content and publishes the final version to our internal documentation system.”article, stated

Evaluation

Agoda evaluated each workflow with operational metrics: alert triage time and human-alignment, phishing precision and false negatives, report drafting time and factual accuracy, plus conservative incident-misclassification monitoring.

Metrics: average alert analysis time · alerts handled per 15 days · human analyst alignment · high-impact incident misclassification · phishing classification precision · false negatives · phishing response time · incident report drafting time · incident report factual accuracy

4 supporting quotes
“The average analysis time per alert has dropped from 20–40 minutes to under 5 minutes, enabling the team to handle over 400 alerts every 15 days more efficiently.”article, stated
“Internal validation over the last three months shows a 97%+ alignment with human analyst conclusions during peer review.”article, stated
“In production, it has maintained over 99% classification precision, correctly identifying both malicious and benign submissions.”article, stated
“In production, this system maintains over 95% factual accuracy and is now part of our regular reporting process.”article, stated

Outcomes

The workflows operate across a global travel platform with millions of users, sensitive data, hundreds of microservices, multiple cloud environments, over 400 alerts per 15 days, and hundreds of weekly phishing reports.

4 supporting quotes
“Tasks that previously consumed significant analyst time, like timeline reconstruction and incident comparison, are now handled end-to-end by the LLM”article, stated
“Since deployment, there have been no known false negatives, and response queues have been virtually eliminated.”article, stated
“This approach has reduced report drafting time to under 10 minutes, with another 30 minutes allocated for review.”article, stated
“LLMs coupled with automations are great assistants in the security incident response domain, especially in lean but highly technical teams.”article, stated