Enterprise-Scale Cloud Event Management with Generative AI for Operational Intelligence

fidelity_investments · Finance · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Cloud operations event intelligence

Fidelity needed to aggregate, enrich, route, and act on cloud health events and support cases across a massive multi-cloud estate, then use generative AI to summarize trends, recommend remediation, and enable agentic operational workflows.

Constraints:

• The cloud estate includes nearly 2,000 AWS accounts and almost 5 million resources.
• Fidelity had aggregated almost 150,000 AWS events in two years.
• Polling did not fit the scale or real-time requirements.
• Raw health events needed business and dependency context for engineers.
• Annual support-case/event analysis could exhaust token limits if sent to on-demand inference.

Architecture

Fidelity built CENTS, an event-driven Cloud Event Notification Transport Service, using AWS Health delegated administration, EventBridge, event enrichment, OpenSearch/search, routing, and contextual delivery. On top, AWS/Fidelity demonstrated MAKI, which sends AWS Health and Support data to Amazon Bedrock, uses on-demand or batch inference depending on volume, summarizes events, identifies categorical buckets and trends, embeds events in OpenSearch with Titan Embeddings, and exposes agent tools for querying health events and related support cases.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  aws-health-delegated-admin["AWS Health delegated admin"] --> eventbridge["EventBridge"]
  eventbridge --> cents-enrichment-pipeline["CENTS enrichment pipeline"]
  cents-enrichment-pipeline --> opensearch-event-store["OpenSearch event store"]
  cents-enrichment-pipeline --> stakeholder-routing["Stakeholder routing"]
  opensearch-event-store --> maki-bedrock-analysis["MAKI Bedrock analysis"]
  maki-bedrock-analysis --> agent-tools["Agent tools"]
  agent-tools --> operational-guidance["Operational guidance"]
  maki-bedrock-analysis --> opensearch-event-store["OpenSearch event store"]

Components

Data flow

AWS Health events aggregate from delegated accounts into EventBridge, enter CENTS for enrichment with Fidelity metadata and dependencies, route to users and OpenSearch, then MAKI augments support/health data and calls Bedrock for categorization, summaries, plans, embeddings, and agentic retrieval of related support cases.

Why this architecture

Fidelity moved to event-driven aggregation because the cloud estate made polling brittle and expensive; MAKI then layered Bedrock analysis on top of clean, enriched event/support data because operational insight depends on consistent data foundations.

Models used: Amazon Bedrock · Amazon Titan Embeddings · Bedrock Converse API · OpenSearch · EventBridge · AWS Health · MAKI

Scale

• About 80% of Fidelity's apps were in the cloud.
  “We got about 80% of our apps in the cloud.”transcript, stated
• The AWS estate included almost 2,000 accounts and almost 5 million resources.
  “Almost 2,000 accounts. Almost 5 million resources.”transcript, stated
• Fidelity aggregated almost 150,000 AWS events in two years.
  “Over almost 150,000 events aggregated in the last two years.”transcript, stated
• The event-driven redesign reduced cost by about 57%.
  “We're reducing our cost by probably 57%.”transcript, stated
• Annual analysis may involve tens of thousands of support cases.
  “A large organization, you may have tens of thousands of support cases.”transcript, stated

Alternatives considered (roads not taken)

Poll for events on a schedule.

Rejected because: Polling became antiquated, costly, and poorly matched to real-time reaction at Fidelity's scale.

Route raw cloud events directly.

Rejected because: Fidelity engineers needed added metadata, dependencies, and context.

Use on-demand Bedrock inference for all analysis.

Rejected because: Large annual analyses could hit token throttles and do not need immediate answers.

Analyze health events without support-case context.

Rejected because: Health events and support cases together give a worldview for actions and impact.

Patterns

async-message-orchestration (orchestration)

CENTS uses event-driven routing over EventBridge rather than scheduled polling.

rag-retrieval (retrieval)

MAKI retrieves event and support-case context from OpenSearch before generating guidance.

batch-offline-processing (optimization)

Large annual analyses are sent to Bedrock batch inference to avoid throttling and reduce cost.

tool-use-orchestration (orchestration)

The MAKI agent uses search tools over health events and support cases to answer operational questions.

Tradeoffs

Move from polling to event-driven processing.

Chose EventBridge and delegated-account event aggregation. over Chron-based polling and pull workflows..

Rationale: Event-driven delivery fit real-time reaction needs and reduced cost.

Use both on-demand and batch Bedrock inference.

Chose On-demand for few streaming events, batch for large retrospective jobs. over One inference mode for all workloads..

Rationale: Urgency and volume differ by event stream versus annual analysis.

Correlate support cases with health events.

Chose One worldview for health and support data. over Separate operational silos..

Rationale: Relevant support cases reveal existing actions for the same health event.

Gotchas (what bit them)

Scale turns simple event collection into a data-foundation problem.

Root cause: Thousands of accounts and millions of resources generate large event volumes.

Resolution: Centralize event aggregation through delegated admin and EventBridge.

Raw events lack the context engineers need to act.

Root cause: AWS event payloads do not include Fidelity-specific metadata and dependencies.

Resolution: Add enrichment before routing events to users.

Large GenAI analyses can hit token throttling.

Root cause: Annual retrospective analysis may involve tens of thousands of cases/events.

Resolution: Route large, non-urgent analyses to Bedrock batch inference.

Evaluation

Fidelity measures the pipeline through cloud coverage, account/resource/event volume, cost reduction, real-time delivery, enriched context usefulness, support-case/event correlation, Bedrock summarization outputs, and batch/on-demand cost behavior.

Metrics: 80% apps in cloud · Almost 2,000 AWS accounts · Almost 5 million resources · Almost 150,000 AWS events in two years · 57% cost reduction · Tens of thousands of support cases for annual analysis

Outcomes

• CENTS aggregated events across nearly 2,000 accounts and almost 5 million resources.
• The event-driven redesign reduced costs by about 57%.
• MAKI produced summaries, categorical buckets, remediation suggestions, and resilience plans from health/support data.
• Agent tooling could retrieve support cases relevant to a specific S3 health event.

Fidelity's estate is multi-cloud, with most apps in the cloud and possible on-prem extension.