AI-Powered Self-Remediation Loop for Large-Scale Kubernetes Operations

salesforce · Tech · 2025 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Large-scale Kubernetes platform operations

Salesforce's Hyperforce Kubernetes team needed to reduce operational toil and improve MTTI/MTTR across a massive multi-cloud Kubernetes fleet where engineers had to manually correlate alerts, metrics, logs, events, traces, runbooks, and siloed internal tools.

Constraints:

the platform managed more than 1,400 clusters and millions of pods
engineers had to investigate thousands of alerts across 50,000 time series and petabytes of logs
existing troubleshooting tools were valuable but siloed and hard to learn
production remediations required strict guardrails, change tracking, rollbackability, and human oversight
runbook quality directly affected RAG retrieval and agent correctness

Architecture

Salesforce implemented a multi-agent self-remediation loop for Kubernetes operations. Alerts land in Slack, where a manager AI agent enriches alert data with RAG-retrieved runbook knowledge and infrastructure topology context, reasons about a troubleshooting plan, and delegates telemetry collection to worker agents connected to Prometheus, K8sGPT, Kube-Magic Mirror, Sloop, Pseudo API, Periscope, and other tools. The manager synthesizes root cause, passes it to a remediation agent, and remediation is executed through safe Argo workflows exposed by an internal compute API, gated by Slack approvals, change management, dashboards, and feedback loops.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack-alert-workspace["slack-alert-workspace"] --> manager-ai-agent["manager-ai-agent"]
  manager-ai-agent --> rag-runbook-store["rag-runbook-store"]
  manager-ai-agent --> topology-context["topology-context"]
  manager-ai-agent --> worker-agent-pool["worker-agent-pool"]
  worker-agent-pool --> prometheus-agent["prometheus-agent"]
  worker-agent-pool --> k8sgpt-agent["k8sgpt-agent"]
  worker-agent-pool --> internal-tools-agents["internal-tools-agents"]
  worker-agent-pool --> manager-ai-agent
  manager-ai-agent --> remediation-agent["remediation-agent"]
  remediation-agent --> human-approval-loop["human-approval-loop"]
  human-approval-loop --> safe-operations-workflows["safe-operations-workflows"]
  safe-operations-workflows --> change-management-and-dashboards["change-management-and-dashboards"]
  human-approval-loop --> slack-alert-workspace

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`slack-alert-workspace`	Collaboration surface for alerts, agent invocation, approvals, feedback, and troubleshooting threads.	`manager-ai-agent`, `human-approval-loop`
`manager-ai-agent`	Chief orchestrator that enriches alerts, plans investigations, delegates data retrieval, and synthesizes RCA.	`slack-alert-workspace`, `rag-runbook-store`, `topology-context`, `worker-agent-pool`, `remediation-agent`
`rag-runbook-store`	Vectorized runbook knowledge used to ground troubleshooting steps and remediation decisions.	`manager-ai-agent`
`topology-context`	Infrastructure and topology knowledge passed as context for reasoning about the platform.	`manager-ai-agent`
`worker-agent-pool`	Specialized agents for telemetry and internal tools, using MCP where available or direct integrations otherwise.	`manager-ai-agent`, `prometheus-agent`, `k8sgpt-agent`, `internal-tools-agents`
`prometheus-agent`	Transforms natural-language metric questions into PromQL/PromptQL-style queries against Prometheus.	`worker-agent-pool`
`k8sgpt-agent`	Retrieves Kubernetes event and realtime pod-log insights.	`worker-agent-pool`
`internal-tools-agents`	Adapters for Kube-Magic Mirror, Sloop, Pseudo API, Periscope, and other Salesforce tools.	`worker-agent-pool`
`remediation-agent`	Chooses proposed safe remediation operations after manager RCA synthesis.	`manager-ai-agent`, `safe-operations-workflows`, `human-approval-loop`
`safe-operations-workflows`	Argo workflows with explicit guardrails for allowed production changes.	`remediation-agent`, `change-management-and-dashboards`
`human-approval-loop`	Slack-based multi-layer approval and feedback gate for production remediation.	`slack-alert-workspace`, `remediation-agent`, `safe-operations-workflows`, `change-management-and-dashboards`
`change-management-and-dashboards`	Audit, visibility, periodic review, and operational tracking for AI-driven actions.	`safe-operations-workflows`, `human-approval-loop`

Data flow

Alerts arrive in Slack. The manager agent retrieves relevant runbook chunks from a RAG vector database and combines them with topology context and alert data. It plans troubleshooting steps, delegates data retrieval to worker agents that call telemetry and internal tooling systems, and synthesizes logs, metrics, events, traces, and runbook context into an RCA. A remediation agent proposes actions such as pod/node restarts, rollout restarts, or configuration changes. Safe operations implemented as Argo workflows enforce guardrails; Slack approvals and change-management tracking control production execution; feedback buttons feed corrections back into agents and runbooks.

4 supporting quotes

“As soon as the alerts land in Slack, the manager AI agent kicks in. It plays the role of the chief orchestrator.”transcript, stated

“runbook knowledge that we had ingested into the RAG vector databases and the infrastructure and topology knowledge”transcript, stated

“delegates the task of gathering the required troubleshooting data to the set of worker agents.”transcript, stated

“Every single operation that we built is an Argo workflow that has the necessary guardrails.”transcript, stated

Why this architecture

Salesforce chose a multi-agent architecture because existing operations knowledge and telemetry were distributed across specialized systems. Reusing those tools let agents connect them meaningfully instead of replacing years of investment. RAG over runbooks and topology gave the manager agent local operational context, while safe operations, Slack approvals, and progressive autonomy kept production actions bounded. The team started with low-risk agents before self-remediation to build confidence and adoption.

4 supporting quotes

“We also made a very conscious decision that we want to reuse all of the existing automation tools”transcript, stated

“So essentially this architecture allowed us to reuse all of our existing tools and the runbooks”transcript, stated

“So when we started looking at the agents, we didn't go big bang from the beginning.”transcript, stated

“progressive autonomy helps, starting with full human oversight to review everything that AI is doing”transcript, stated

Models used: AWS Bedrock multi-agent collaboration · LangChain · LangGraph · Prometheus · K8sGPT · Argo CD · Argo Workflows · Slack · MCP

Scale

Salesforce described managing more than 1,000 Kubernetes clusters as the target class of problem.
“how it feels to manage 1,000-plus Kubernetes cluster, folks.”transcript, stated
On-call engineers may face thousands of alerts.
“there are thousands of alerts waiting to be attended”transcript, stated
Troubleshooting can involve 50,000 time series metrics and two petabytes of logs.
“skim through 50,000 times series of metrics, two petabytes of logs.”transcript, stated
A five-minute fix could take five hours to troubleshoot.
“When a five-minute fix takes you five hours to troubleshoot”transcript, stated
Hyperforce manages more than 1,400 clusters across multiple substrates.
“We manage more than 1,400 clusters across multiple substrates.”transcript, stated
The platform runs hundreds of thousands of compute nodes and millions of pods.
“We run hundreds of thousands of compute nodes and our platform scales millions of pods today.”transcript, stated
The team supports 40-plus operators, controllers, and integrations.
“We also provide support for 40-plus operators, controllers, and integrations.”transcript, stated
More than a thousand hours were spent on support.
“more than a thousand hours is spent on support.”transcript, stated
Salesforce projected 5X platform growth over the next couple of years.
“We are looking at 5X growth for our platform in the next couple of years.”transcript, stated
The on-call report agent automated about 90% of manual report work.
“It automated about 90% of the work that an engineer would do.”transcript, stated
Live-site analysis previously took engineers at least a couple of days every week.
“The engineers were spending at least a couple of days every week.”transcript, stated
The self-remediation loop improved troubleshooting time by 30% and saved almost 150 hours a month.
“It improved by 30%.”transcript, stated
“saved almost 150 hours a month, which roughly equates to one person or one engineer worth of bandwidth”transcript, stated

Alternatives considered (roads not taken)

manual cross-tool troubleshooting

Rejected because: Engineers had to log into multiple systems, pass context manually, and learn many siloed tools.

“These tools are pretty much siloed. They don't really talk to each other.”transcript, stated

“Every engineer on the team needs to learn all of these tools and also know how to effectively use them, how and when to effectively use”transcript, stated

big-bang self-healing rollout

Rejected because: Salesforce started with lower-risk agents to understand capabilities and build trust before production remediation.

“we didn't go big bang from the beginning. We chose to start small. We wanted to first understand what these agents are capable of and how best to leverage them.”transcript, stated

“And we also clearly saw the potential that we could apply agents to implement a self-healing loop in production, and which can greatly improve our production support problem.”transcript, stated

letting agents use raw kubectl or cloud SDK operations freely

Rejected because: Unbounded access could delete control planes, applications, or node pools and cause outages.

“If we let AI make changes at will in production, it can be really catastrophic.”transcript, stated

“AI can choose to delete your Kubernetes control plane, can choose to delete your application or your node pool. And all of them can result in outages.”transcript, stated

runbook-only reasoning for all complex failures

Rejected because: Runbooks struggle to connect causal chains across pods, CoreDNS, nodes, NICs, and noisy neighbors at fleet scale.

“our runbook-based solutions struggles to really connect the dots. Like, this is a good example of it. Like, assume that there is an application”transcript, stated

“It is very difficult to write a runbook that solves this problem. Even if we manage to write this, it is very specific and for a very single use case.”transcript, stated

Patterns

`multi-agent-orchestration` (agent-architecture)

A manager agent orchestrates worker agents and a remediation agent to gather telemetry, reason over root cause, and propose safe actions.

“there is an agent of the agents which can communicate with individual agents.”transcript, stated

“The worker agents specialize in talking to various data systems across the infrastructure”transcript, stated

`rag-retrieval` (knowledge-grounding)

Runbook knowledge is ingested into RAG vector databases and retrieved to guide troubleshooting plans and RCA synthesis.

“runbook knowledge that we had ingested into the RAG vector databases”transcript, stated

`tool-use-orchestration` (agent-architecture)

Agents reuse Prometheus, K8sGPT, Sloop, Pseudo API, Periscope, Kube-Magic, and Argo workflows as operational tools.

“we built specific worker agents that talks to KMM, Sloop, Pseudo API, Periscope, and all of the other tools”transcript, stated

`human-in-the-loop-gate` (reliability)

Production remediation requires Slack approval, including an added second approval layer, before safe operations execute.

“we implemented a Slack-based human in the loop approval process.”transcript, stated

“after the on-call engineer clicks yes, it goes to another engineer or a manager who has to approve”transcript, stated

`guardrails-io-filtering` (safety)

Allowed operations are wrapped as guarded Argo workflows that enforce safe restart, scale, and rollout constraints.

“Every single operation that we built is an Argo workflow that has the necessary guardrails.”transcript, stated

“AI cannot do anything that we do not allow it to.”transcript, stated

`observability-tracing` (operations)

AI-driven operations are tracked through change management, dashboards, periodic review, and Slack feedback.

“strict change management processes and auditing controls”transcript, stated

“We also implemented Slack-based feedback loop.”transcript, stated

Tradeoffs

Reuse existing internal operations tools.

Chose worker agents over KMM, Sloop, Pseudo API, Periscope, and related tooling over building a new standalone AI operations stack.

Rationale: Reusing mature tooling produced the quickest wins while preserving domain-specific operational knowledge.

“leveraging existing tools helped a lot. It was the quickest and the biggest wins that we had.”transcript, stated

“We also made a very conscious decision that we want to reuse all of the existing automation tools”transcript, stated

Use Slack as the operations control plane.

Chose Slack alerts, approval, feedback, collaboration, and on-demand queries over a separate AI operations dashboard.

Rationale: Slack already hosted collaboration and made it easy to implement approvals, feedback, summaries, and alert-triggered agent runs.

“Slack is the space where we collaborate and track the progress of the investigations.”transcript, stated

“it made it very easy for us to implement the approval, the multi-layer approval processes and the feedback loops.”transcript, stated

Start with full human oversight and relax gradually.

Chose progressive autonomy over immediate autonomous production remediation.

Rationale: The team wanted confidence that actions were safe before granting agents more independent production authority.

“To begin with, no operation can happen in production without human approvals.”transcript, stated

“As we gain more and more confidence with AI, we can relax that constraints and give more autonomy to AI over time and let it take actions one by one and let it, so we can allow AI agents to make more autonomous actions one by one as we gain more confidence.”transcript, stated

Gotchas (what bit them)

Existing tools did not solve toil because they were siloed.

Root cause: Troubleshooting required manual context transfer and broad tool expertise across the team.

Resolution: Wrap the tools in worker agents and have the manager agent connect their outputs into a coherent RCA.

“We have to manually pass context between them to troubleshoot issues.”transcript, stated

“AI agents were able to connect all of these tools meaningfully”transcript, stated

Runbook quality controlled agent success.

Root cause: Duplicate, conflicting, or poorly structured runbooks hurt RAG chunking, retrieval, and decision quality.

Resolution: Define a runbook strategy for creation, modification, structure, and ongoing updates.

“The structure of the runbook and the accuracy of the runbook determines the success of this overall architecture.”transcript, stated

“If you have duplicate runbooks with conflicting information, we can't expect AI to make right decisions.”transcript, stated

Direct production actions by AI can cause outages.

Root cause: Unbounded operations, missing guardrails, poor visibility, and over-autonomy make production remediations risky.

Resolution: Limit the operation set, encode seasoned-engineer guardrails, track all changes, and use progressive autonomy.

“If we let AI make changes at will in production, it can be really catastrophic.”transcript, stated

“coded them as guardrails within every single operation that we allowed AI to take.”transcript, stated

Complex causal chains exceeded simple runbook traversal.

Root cause: Problems can span application latency, CoreDNS, node networking, NIC bandwidth, and noisy pods across different resources.

Resolution: Explore infrastructure knowledge graphs so AI can traverse relationships and failure propagation.

“How can we do this meaningfully across the entire scale of the infrastructure for all kinds of problems”transcript, stated

“we are attempting to solve this using knowledge graphs.”transcript, stated

Evaluation

Salesforce measured operational impact through troubleshooting-time improvement, support hours saved, adoption of earlier agents, and feedback loops from Slack approvals/corrections into runbooks and agents.

Metrics: 30% improvement in troubleshooting time · almost 150 hours saved per month · on-call report agent automated about 90% of manual work · live-site analysis saved at least a couple of days per week · target to eliminate 80% of production support toil

3 supporting quotes

“It improved by 30%.”transcript, stated

“saved almost 150 hours a month, which roughly equates to one person or one engineer worth of bandwidth”transcript, stated

“It automated about 90% of the work that an engineer would do.”transcript, stated

Outcomes

the self-remediation loop ran in production with guarded human-approved actions
troubleshooting time improved by 30%
agentic operations saved almost 150 hours per month
early agents for reports, kubectl, and live-site analysis gained adoption and reduced toil
Salesforce identified knowledge graphs and success/failure memory as next areas for deeper autonomy

The strongest current metrics are 1,400+ clusters, millions of pods, 5X expected growth, 30% troubleshooting improvement, and almost 150 hours/month saved.