Building Agentic AI Assistant for Observability Platform

grafana · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Observability platform assistance

Grafana Labs needed an assistant that could help users query observability data, change dashboards, investigate incidents, and learn Grafana while operating inside the user's current Grafana context.

Constraints:

Grafana has many features, data sources, tools, and workflows, so context-window and tool-selection pressure is high.
Prompts were coupled to model versions such as Claude 3.5 versus 3.7.
The team needed fast iteration in a changing LLM landscape without overengineering too early.
Human oversight and traceability mattered because the assistant can query data and modify dashboards.

Architecture

Grafana built an agentic chat assistant that runs in the browser as a sidebar and performs tool calling in the frontend. The assistant uses Claude, Grafana context, tool descriptions/schemas, error feedback loops, natural-language tool responses, tool overloading for workflow-like operations, and visible intermediate steps so users can inspect and guide investigations.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  grafana-user[Grafana User] --> assistant-sidebar[Assistant Sidebar]
  assistant-sidebar --> browser-agent-loop[Browser Agent Loop]
  assistant-sidebar --> grafana-context[Grafana Context]
  grafana-context --> browser-agent-loop
  browser-agent-loop --> claude[Claude]
  browser-agent-loop --> grafana-tools[Grafana Tools]
  grafana-tools --> tool-error-feedback[Tool Error Feedback]
  tool-error-feedback --> browser-agent-loop
  grafana-tools --> grafana-data-sources[Grafana Data Sources]
  grafana-tools --> dashboards[Dashboards]
  dashboards --> grafana-user

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Grafana User`	Person asking questions, changing dashboards, investigating incidents, or learning Grafana.	`Assistant Sidebar`
`Assistant Sidebar`	Agentic chat UI that follows the user around their Grafana stack.	`Grafana User`, `Browser Agent Loop`, `Grafana Context`
`Browser Agent Loop`	Client-side agent loop and tool-calling runtime.	`Assistant Sidebar`, `Claude`, `Grafana Tools`, `Tool Error Feedback`
`Claude`	Primary LLM provider/model family used for the assistant.	`Browser Agent Loop`
`Grafana Context`	Current stack/page/dashboard/data-source context available because the assistant runs inside Grafana.	`Assistant Sidebar`, `Browser Agent Loop`
`Grafana Tools`	Query, dashboard editing, investigation, data-source, and validation tools exposed to the assistant.	`Browser Agent Loop`, `Grafana Data Sources`, `Tool Error Feedback`, `Dashboards`
`Tool Error Feedback`	Errors returned to the LLM so it can repair bad queries or tool inputs.	`Browser Agent Loop`
`Grafana Data Sources`	Observability backends such as Prometheus/Loki and other Grafana-supported sources.	`Grafana Tools`, `Dashboards`
`Dashboards`	Visualizations and panels the assistant can query or modify.	`Grafana Tools`, `Grafana User`

Data flow

A user opens the sidebar inside Grafana, the assistant receives current page/context plus its system prompt and tools, calls browser-side tools to query data or modify dashboards, feeds tool results and errors back to the LLM for iterative repair, and displays intermediate data/steps so the human can keep oversight.

3 supporting quotes

“basically and it's an agentic chat. So just what you described it's a lot of for loops iterating one after another and you can chat with it in using graphana in in Graphana cloud. Uh so it always kind of follows you around your gapana stack and it's always there to help you.”transcript, stated

“is that it it's the agent bit runs in the browser and certainly all the tool calling happens in the browser and basically the first version we built came from a hackathon project and so you have a week to basically build something or tell a story actually and thanks to”transcript, stated

“just passing the error back to the LLM will actually help the the LLM to figure out the right inputs or how to use the tool”transcript, stated

Why this architecture

Grafana kept the first architecture simple and frontend-first because the hackathon needed a working proof quickly, the assistant benefits from being inside the user's current Grafana context, and overbuilding multi-agent architectures before learning from real use would be risky.

3 supporting quotes

“thing I think you said like let's just build it in the front end um don't have any back end like keep it dead simple um and just use the LLM really as the back end that was a good choice wasn't it >> yeah uh yeah I guess it's probably the first lesson that we could give here or”transcript, stated

“graphana in in Graphana cloud. Uh so it always kind of follows you around your gapana stack and it's always there to help you. Um and you basically can can do a lot of things that were very very hard to do before. For example, changing single colors in a dashboard or I don't know”transcript, stated

“like try to build a small P to prove maybe first without you know it's doesn't have to be perfect Um, and you know, actually, if you're trying to get it perfect from the start, you're probably going to overengineer uh things that maybe will never get used by the”transcript, stated

Models used: Claude 3.5 · Claude 3.7 · Claude · Gemini

Scale

The initial assistant came from a one-week hackathon project.
“basically the first version we built came from a hackathon project and so you have a week to basically build something or tell a story actually and thanks to curser and the fact that you you know you you are both very um talented engineers as well but we're able to actually build something real.”transcript, stated
The assistant's system prompt and tools consumed about 40k tokens together at one point.
“the system prompt itself I think it's uh I think it's 20k uh token which is quite which is a lot uh and the tools are also 20k so we are at 40k”transcript, stated
The team discussed million-token context windows but cautioned that larger context does not automatically solve complex workflows.
“think the 1 million like type of context window that we we're seeing now in cloud and and Gemini are awesome for uh you know tasks that are dealing with massive data set rather than mass complex uh workflow because the the the intelligence actually start to fall off”transcript, stated
Grafana's feature surface spans areas like IRM, alerting, data sources, and dashboards.
“graphana uh feature wise it's big like you have like IRM you have alerting each data source dashboarding is massive”transcript, stated

Alternatives considered (roads not taken)

Start with a fully designed backend system

Rejected because: The team wanted a small proof first and the LLM tooling space was changing too quickly for a months-long design phase.

“I don't think it's appropriate to just sit back and design a big system for months and then slowly build this enormous thing.”transcript, stated

Keep LangChain TypeScript in the browser

Rejected because: The team hit limitations and wanted more flexibility/control than the framework provided.

“are working we we we realized that you know we will have more flexibility by not uh using line chain. Um I think a big thing on chain is also having like obsability uh and and there's like other other features that are probably very good but we were not using. So this is”transcript, stated

Expose every possible Grafana action as a separate tool

Rejected because: Too many tools confuse the model and consume too much context.

“If you have like thousands of tools, the agent might not know which one to pick or which one to use or how to use it even.”transcript, stated

Swallow tool errors

Rejected because: Returning errors lets the LLM correct syntax, inputs, and tool use.

“bit about tools here and one of the the thing that I think is the one of the most important ones is to not swallow error messages or errors that are thrown in in tool calls. I think we we did that in the beginning sometimes and but we realized that like just passing the”transcript, stated

Use multi-agent architecture immediately

Rejected because: The team considered multi-agent complexity, context handoff, telephone-game effects, and multiple voices too costly without a strong reason.

“literally saying like don't use multi- aent because it's quite complex. >> I think what they're trying to say is that you probably need a good reason for building multi- aent like don't just build it because you think you need it”transcript, stated

Patterns

`tool-use-orchestration` (agent architecture)

The assistant loops over browser-side tool calls for querying, dashboard changes, and investigations.

“make tool calls and do things in response to those, you know, what it gets back from those tool calls.”transcript, stated

`observability-tracing` (observability)

The assistant itself operates inside an observability platform and exposes investigation steps/data so users can trace work.

“during an incident you want to be able to trace back how did we came to this conclusion”transcript, implied

“I think something that we also always wanted is to keep the human in a loop”transcript, implied

`context-compaction` (context management)

Tool overloading and workflow routing are used to reduce tool/context footprint in a large product surface.

“tool overloading is the idea of like combining multiple tools uh into a single one”transcript, stated

`prompt-versioning` (prompt operations)

The team learned prompts were coupled to Claude model versions and provider-specific behavior.

“noticed Claude between 35 and 37 our prompts didn't perform as well because we kind of fine-tuned them”transcript, stated

`human-in-the-loop-gate` (governance)

The assistant shows steps and data so users remain in control rather than delegating observability work invisibly.

“we have uh you know all this these great visualizations that we can use in our tool belt”transcript, implied

`eval-driven-development` (quality)

The team explicitly discusses evaluation as test-like guardrails because adding tools can break behavior.

“evaluation a bit like a bit like test”transcript, stated

“then you don't necessarily realize that you you're breaking something. So evaluation are definitively great ways”transcript, stated

Tradeoffs

Prototype architecture

Chose Browser-first agent loop over Backend-first production architecture.

Rationale: The browser context let the team prove usefulness quickly in a hackathon while keeping the assistant close to user context.

“the first lesson that we could give here or the first uh um you know um advice is like try to build a small P to prove maybe first”transcript, stated

Framework dependency

Chose Remove LangChain for custom control over Keep a general agent framework in the browser.

Rationale: The team wanted intuitive control and freedom from framework assumptions.

“building something you're in more control you understand it intuitively and you can change it.”transcript, stated

Tool response format

Chose Natural-language tool responses over JSON, stringified JSON, or XML responses.

Rationale: The LLM performed better with natural-language descriptions of tool results.

“we tried returning JSON or stringified JSON or we tried returning XML, but we figured out that just returning natural language is actually better”transcript, stated

Tool grouping

Chose Overloaded workflow-like tools as an interim solution over Separate tool for every data-source operation.

Rationale: Combining operations reduced context/tool count but the team acknowledged it was a hacky midpoint.

“it's a quick fix that has allowed us to uh solve the problem of like having too many tools”transcript, stated

Gotchas (what bit them)

Prompts can become tied to a specific model version.

Root cause: Prompts optimized for Claude 3.5 behaved differently on Claude 3.7 or other providers.

Resolution: Treat model changes as prompt-engineering events, not drop-in swaps.

“they they are tied or coupled to the model itself, didn't we?”transcript, stated

Too many tools confuse agents and consume context.

Root cause: Grafana's product surface has many data sources and actions, multiplying tool descriptions.

Resolution: Start broad to learn, then narrow workflows, overload related tools, or route to specialized workflows.

“if you add too many then then going to get confused but on the other side if you don't like if you add not enough tool then you're losing in creativity”transcript, stated

Swallowed tool errors prevent self-correction.

Root cause: The LLM cannot repair bad inputs or syntax if it never sees the error.

Resolution: Return tool errors to the model so it can iterate like a person debugging.

“feeding the result, but even if it's a negative result back into the agent so it can keep going and try try again.”transcript, stated

Multi-agent architectures add context-handoff and voice consistency problems.

Root cause: Multiple agents create telephone-game effects and may speak/act as separate voices.

Resolution: Prefer a single agent or workflow-style tools until there is a strong reason to split.

“you have the question of like telephone game and how do I pass the the right context to one agent from another”transcript, stated

Evaluation

Grafana evaluates the assistant through hackathon/user adoption, tool-use behavior, prompt/model-version comparisons, error-feedback loops, workflow/tool changes, and explicit evaluation tests as tool additions risk regressions.

Metrics: Tool-call success · Query/dashboard task completion · Prompt behavior by model version · Context/tool token budget · Evaluation regression failures · Human-visible investigation steps

“you have to do it. You have to build it. You have to wire it up and try it.”transcript, stated

“our architecture and our um like some of the approaches that we've taken have evolved and changed quite a lot”transcript, stated

Outcomes

Grafana shipped an assistant sidebar that can answer questions, query data, and change dashboards.
The team found client-side prototyping effective and later hardened architecture decisions.
The assistant design surfaced concrete lessons around prompt/model coupling, tool count, error feedback, natural-language tool responses, and multi-agent complexity.

The transcript is a design discussion rather than a metrics-heavy launch report; it reports architecture and lessons more than adoption volume.

“you basically can can do a lot of things that were very very hard to do before.”transcript, stated

“you can query as as you just said Matt but then you can based on this query do another investigation or create a dashboard”transcript, stated