Multi-Agent AI Systems for IT Operations and Incident Management

kolomolo_/delaval/_arelion · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: IT operations and incident management

Kolomolo needed to automate two high-volume operational workflows: incident triage across DeLaval's connected dairy-farm equipment and maintenance-notification processing across Arelion's global network vendor emails.

Constraints:

• The DeLaval environment spanned thousands of connected farms, long-lived industrial equipment, and multiple monitoring systems.
• Arelion received maintenance notices from more than 100 vendors in inconsistent formats, languages, and structures.
• The systems had to be cost efficient enough for production volumes, so not every task could go through a frontier LLM.
• Operators and stakeholders needed traceability, confidence scores, and human review paths for low-confidence or risky decisions.

Architecture

Kolomolo implemented two related production AI systems. DeLaval's Unity Ops is a serverless AWS multi-agent incident system in which monitoring alerts trigger specialized agents for formatting, RAG over playbooks and Jira history, log/code/CMDB analysis, and Jira ticket creation. Arelion's pipeline uses classical ML for high-volume email classification, then routes maintenance notifications to Bedrock-backed extraction with prompt optimization and human review for low-confidence cases.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  monitoring-sources["Monitoring Sources"] --> aws-event-layer["AWS Event Layer"]
  aws-event-layer --> lambda-agent-orchestrator["Lambda Agent Orchestrator"]
  lambda-agent-orchestrator --> knowledge-base-agent["Knowledge Base Agent"]
  knowledge-base-agent --> vector-stores["Vector Stores"]
  lambda-agent-orchestrator --> diagnostic-agents["Diagnostic Agents"]
  diagnostic-agents --> jira-agent["Jira Agent"]
  lambda-agent-orchestrator --> jira-agent
  lambda-agent-orchestrator --> dynamodb-trace-store["DynamoDB Trace Store"]
  dynamodb-trace-store --> observability-dashboard["Observability Dashboard"]
  email-classification-layer["Email Classification Layer"] --> entity-extraction-layer["Entity Extraction Layer"]
  entity-extraction-layer --> change-management-tool["Change Management Tool"]

Components

Data flow

For DeLaval, an alert from monitoring infrastructure enters an AWS event path, triggers Lambda-hosted agent orchestration, gathers alert details, retrieves playbooks and similar historical tickets, lets diagnostic agents contribute context, and finally creates an enriched Jira incident. For Arelion, incoming vendor emails are vectorized and classified, low-confidence cases go to human review, and maintenance notifications continue to LLM entity extraction before being inserted into the change-management tool.

Why this architecture

Kolomolo used multi-agent orchestration where diagnosis needed several specialist viewpoints, but used a hybrid ML/LLM path for email processing because classification volume made all-LLM processing too expensive. The common design principle is to reserve expensive generative calls for tasks needing language flexibility while keeping predictable routing, confidence scoring, and auditability around them.

Models used: AWS Bedrock foundation models · XGBoost · classical vector classifier

Scale

• DeLaval's current monitoring scope was about 3,000 connected farms.
  “The system currently monitors approximately 3,000 connected farms with plans to scale to 7,000 by the following year and 20,000-30,000 farms within five years.”summary, stated
• Arelion's vendor notification flow involved more than 100 vendors.
  “The operational challenge stemmed from the constant stream of maintenance notifications from over 100 different vendors.”summary, stated
• The Arelion network had global backbone scale.
  “Arelion operates one of the world's largest internet backbone networks spanning from Mexico to Singapore, with approximately 70% of global internet traffic touching their infrastructure”summary, stated
• The email classifier reported about 95% accuracy with sub-second processing.
  “This approach achieved approximately 95% accuracy for classification with sub-second processing time.”summary, stated
• The LLM extraction stage reported about 97% accuracy.
  “The system achieved approximately 97% accuracy in entity extraction with confidence scores, and automatically handled the dynamic adaptation to different email formats”summary, stated

Alternatives considered (roads not taken)

Use a pure LLM for all email classification and extraction

Rejected because: The presentation argues that frontier LLM classification would be economically unsustainable at Arelion's email volumes.

Use rule-based parsing for vendor maintenance notifications

Rejected because: Rules would be brittle across vendor formats, languages, and minor template changes.

Keep incident response as a human coordination process

Rejected because: The existing process forced people to route and dispute issues manually across thousands of farm deployments.

Use a rigid workflow for all root-cause analysis

Rejected because: Kolomolo kept static flow for known steps but used dynamic group chat when the solution path was not predetermined.

Patterns

multi-agent-orchestration (architecture)

Unity Ops decomposed incident response into specialized agents and used both static workflow and dynamic group chat patterns for root-cause analysis.

rag-retrieval (knowledge)

The Knowledge Base Agent retrieved playbooks and similar Jira tickets so current incidents could be enriched with past resolutions and root causes.

confidence-thresholding (operations)

Arelion used classifier confidence scores to route uncertain maintenance-email decisions to a manual review queue.

observability-tracing (operations)

Kolomolo logged conversations, traces, tokens, and decisions, then exposed agent debates through a custom dashboard to build stakeholder trust.

cost-aware-routing (optimization)

The Arelion design combined cheap classical classification with selective LLM extraction and model routing by task complexity.

Tradeoffs

Use hybrid ML plus LLM processing for vendor emails

Chose Classical ML classification followed by selective LLM entity extraction over A pure LLM pipeline for all incoming emails.

Rationale: This kept cost and latency manageable while preserving LLM flexibility for varied extraction tasks.

Use both static and dynamic agent conversations

Chose Static workflow for known incident steps plus dynamic group chat for uncertain root-cause analysis over A single rigid workflow for every incident.

Rationale: Routine ticket enrichment benefits from predictable sequencing, while root-cause analysis benefits from specialist debate.

Use serverless AWS infrastructure for incident automation

Chose Lambda, SNS/API Gateway, DynamoDB, and CDK over A manually scaled always-on service stack.

Rationale: The event-driven operational workload needed automatic scaling, reproducible deployment, and cost efficiency.

Gotchas (what bit them)

Pure rule-based email processing could not survive vendor format drift.

Root cause: Arelion's suppliers sent maintenance notices in many formats, languages, and structures, making brittle matching rules impractical.

Resolution: Kolomolo used ML classification plus LLM extraction, with prompt optimization to adapt as vendor formats changed.

Agentic incident automation required transparency before stakeholders would trust it.

Root cause: Black-box ticket creation would not explain why agents reached conclusions or how they debated root causes.

Resolution: The team logged agent conversations and built a dashboard showing decisions, token use, and debates.

Future autonomous code remediation adds production risk even with approval gates.

Root cause: Moving from analysis to automated merge requests could introduce critical bugs or security vulnerabilities.

Resolution: The roadmap explicitly kept CI/CD guardrails and senior engineer approval before deployment.

Evaluation

The case reports operational accuracy, efficiency, cost, confidence scoring, and observability rather than a formal benchmark harness.

Metrics: approximately 95% classification accuracy · approximately 97% entity extraction accuracy · 80% reduction in SRE reactive ticket handling · 80% reduction in manual classification workload · claimed 10,000% reduction in classification costs

Outcomes

• DeLaval automated incident enrichment for connected farm operations and reduced reactive SRE handling time.
• Arelion reduced manual maintenance-notification classification work and improved adaptation to changing vendor formats.
• Both systems demonstrated production LLMOps patterns around observability, cost routing, human review, and continuous adaptation.

The case reports DeLaval at about 3,000 connected farms with planned growth to tens of thousands, and Arelion spanning a backbone network touched by a large share of global internet traffic.