Multi-Agent AI System for Network Change Management

cisco · Telecommunications · year n/a · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Network operations and production change management

Cisco Outshift built a multi-agent system to reduce production failures during network change management by helping operations teams assess impact, generate test plans, run validation tests, and attach evidence back to ITSM workflows.

Constraints:

• Network environments contain multiple vendors, devices, controllers, agents, configuration systems, SIEM data, telemetry, YANG, JSON, and other formats.
• Network change approvals happen through enterprise ITSM workflows, so agent outputs need to attach back to tickets for approval boards.
• Agents need a network representation they can query predictively and meaningfully, rather than raw heterogeneous device data.
• The system is at MVP stage and has to keep learning from customer use while supporting production-environment testing.

Architecture

Cisco's system combines a natural-language/ITSM interface, an assistant planner agent, four specialized ReAct-style agents, and a knowledge-graph-based digital twin of the production network. An ingestion service normalizes controllers, Splunk/SIEM, traffic telemetry, device data, and configuration files into OpenConfig and stores them in a layered graph, implemented with ArangoDB for the current use case. Agents use the graph and test-execution tools such as Batfish or Routenet to assess impact, generate test plans, execute validation against GitHub change candidates plus live graph snapshots, and attach reports back to ServiceNow tickets.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ITSM["ServiceNow ITSM"] --> NL["Natural Language Interface"]
  NL --> Assistant["Assistant Agent"]
  Assistant --> Impact["Impact Assessment Agent"]
  Assistant --> Plan["Test Plan Agent"]
  Assistant --> Query["Query Agent"]
  Assistant --> Executor["Executor Agent"]
  Sources["Production Network Sources"] --> Ingestion["Network Data Ingestion Service"]
  Ingestion --> OpenConfig["OpenConfig Schema"]
  OpenConfig --> Graph["Knowledge Graph Digital Twin"]
  Query --> Graph
  Impact --> Graph
  Plan --> Graph
  GitHub["GitHub Change Repository"] --> Executor
  Executor --> Graph
  Executor --> Tools["Test Execution Tools"]
  Graph --> Tools
  Executor --> ITSM
  Framework["Open Agent Framework"] --> Assistant
  Framework --> ITSM

Components

Data flow

A network engineer submits a change request in ServiceNow. Cisco's application ingests the ticket in natural language and an assistant agent plans follow-on work. Specialized agents summarize the request, create an impact assessment, attach that assessment to the ITSM ticket, generate test cases and expected results, and attach the test plan back for approval. When a change candidate is available in GitHub, the executor agent reads the pull request, snapshots the latest network state from the knowledge graph, combines the two inside the digital twin, runs network tests one by one with engineering tools, and writes pass/fail results plus remediation recommendations back to the ticket.

Why this architecture

Cisco used agents where the change-management workflow required interpretation, impact assessment, test-plan reasoning, and validation rather than simple rule automation. The knowledge graph/digital twin gives agents a structured representation of heterogeneous network data, while OpenConfig makes the schema legible to LLMs. The layered graph keeps tests from traversing irrelevant data, and the ITSM integration keeps generated assessments and reports inside the existing approval path.

Models used: Unspecified LLM-powered ReAct agents · Fine-tuned query agent for ArangoDB/AQL graph queries

Scale

• The system is an MVP delivered into customer production environments through Cisco's incubation process.
  “we come up with prototypes we do a testing b testing and then we kind of deliver an MVP into a production environment”transcript, stated
• Cisco describes five agents in the current application.
  “we delivered this set of agents. Uh we built a set of agents as a group. So we have five agents right now as part of this application.”transcript, stated
• The graph has to cover multiple vendors, firewalls, switches, routers, controllers, device agents, configuration management systems, SIEM, and telemetry.
  “variety of vendors in the customers environment, variety of devices, firewalls, switches, routers and so on.”transcript, stated

  “controllers systems there the devices themselves there agents in the devices there are configuration management systems all of these things”transcript, stated

Alternatives considered (roads not taken)

Rule-based automation for the change-management problem

Rejected because: Cisco evaluated the workflow and found places where AI agents added value beyond rules, especially around impact assessment and test planning.

RAG for querying the knowledge graph

Rejected because: The RAG attempt did not work well because AQL queries traversed all graph layers, burning many tokens and taking too long.

Neo4j as the graph database

Rejected because: Neo4j was a market-leader candidate, but the team selected ArangoDB for the current work due to historical use in related Cisco recommendation-system cases, while still exploring Neo4j.

Bespoke point-to-point agent integrations

Rejected because: Cisco wants open agent interoperability across MCP, A2A, identity, schemas, directories, composition, and observability instead of rebuilding integrations for every agent pair.

Patterns

multi-agent-orchestration (architecture)

The application uses an assistant/planner agent plus specialized ReAct agents for impact assessment, test planning, graph querying, and execution.

tool-use-orchestration (architecture)

Agents call ITSM, GitHub, the knowledge graph, and network-engineering test tools to move from ticket to validation report.

rag-retrieval (architecture)

The knowledge graph includes vector indexing for semantic search and graph-RAG-style retrieval, although the graph query agent was ultimately fine-tuned for AQL.

model-routing (architecture)

Cisco fine-tuned the query agent for graph access after RAG-based querying performed poorly.

human-in-the-loop-gate (safety)

Agent-generated impact assessments and test plans are attached to the ITSM ticket so the approval board can review before production implementation.

observability-tracing (operations)

The open agent framework includes observability for agent interactions and production operation.

eval-driven-development (quality)

Cisco measures agents, the knowledge graph, and digital twin with customer-use-case-linked extrinsic metrics while the MVP evolves.

Tradeoffs

Use AI agents for selected workflow steps instead of only rule-based automation.

Chose Agents for impact assessment, testing, reasoning, and validation over Pure rule-based change automation.

Rationale: Cisco found parts of the change-management workflow where reasoning agents could reduce customer pain.

Standardize network data on OpenConfig.

Chose OpenConfig schema over Leaving network data in vendor-specific formats.

Rationale: OpenConfig consolidates heterogeneous data and is well documented enough for LLMs to understand.

Select ArangoDB for the current graph implementation.

Chose ArangoDB over Neo4j as the immediate graph database choice.

Rationale: Historical Cisco use in security recommendation systems outweighed Neo4j's market-leader position for this implementation.

Fine-tune the query agent.

Chose Fine-tuned graph query agent with schema and example queries over RAG-based graph querying.

Rationale: Fine-tuning reduced token burn and response time after RAG caused broad graph-layer traversal.

Use layered graph access rather than always traversing the full network graph.

Chose Layer-specific graph queries depending on test intent over Every query traversing all graph layers.

Rationale: Configuration drift tests can hit raw configuration layers, while reachability tests need raw configuration, data plane, and control plane layers.

Gotchas (what bit them)

Network data could not be handed to agents in raw heterogeneous forms.

Root cause: Controllers, devices, SIEM, telemetry, and configuration systems emit YANG, JSON, streaming telemetry, and other formats across many vendors and device classes.

Resolution: Cisco built an ETL ingestion service that normalizes network data into OpenConfig and stores it in a layered graph.

Graph RAG caused expensive and slow knowledge-graph queries.

Root cause: AQL queries traversed all graph layers during reasoning loops, burning tokens and delaying results.

Resolution: Cisco fine-tuned the query agent with schema information and example queries.

Manual change planning often misses the right tests to run.

Root cause: Network teams run many tests but may omit important cases for the specific change intent.

Resolution: A test-plan agent reasons through available test-plan information and change intent, then lists test cases and expected results.

Agent-to-agent integration can become bespoke and expensive.

Root cause: Connecting agents from different systems usually requires reconstructing or custom-integrating agents each time.

Resolution: Cisco participates in an open framework covering identity, skill schemas, directories, composition, observability, MCP, and A2A.

Evaluation

Cisco evaluates the agents, knowledge graph, digital twin, and customer value using extrinsic metrics tied to the customer use case rather than abstract intrinsic graph metrics.

Metrics: Agent performance and multi-agent coordination. · Knowledge graph usefulness for the customer use case. · Digital twin effectiveness for network validation. · Token consumption and response time improvements after query-agent fine-tuning.

Outcomes

• Cisco delivered an MVP into customer environments for network change-management validation.
• The query-agent fine-tuning reduced token consumption and result latency compared with the RAG-based graph-query attempt.
• The system can summarize change requests, generate impact assessments, generate test plans, run validation in a digital twin, and attach reports back to ITSM tickets.
• Cisco identified the knowledge graph and open agent framework as critical building blocks for scalable customer solutions.

The MVP handles heterogeneous multi-vendor network environments and currently uses five agents in the application.