Building Alex: An AI Agent for LLM Observability and Debugging

arize_ai · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: LLM observability and debugging assistant

Arize wanted Alex to help users debug, optimize, and learn its observability platform without needing expert solutions architects for every workflow.

Constraints:

Customers needed guidance through complex traces, evals, monitors, prompts, and platform workflows.
Early LLM capabilities favored structured/on-rails workflows.
The team had a November 2023 start and a July 2024 launch deadline.
Evaluation had to cover tool calls, tasks, sessions, and traces rather than a single generic score.

Architecture

Alex began as an on-rails decision-tree copilot using OpenAI tool calling and solutions-architect playbooks, then evolved toward more autonomous workflows as models improved. Arize used its own platform to inspect traces, define evals at multiple levels, dogfood skills behind feature flags, and convert expert debugging workflows into templates and agent tasks.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user[User] --> alex-copilot[Alex Copilot]
  alex-copilot --> tool-calling-router[Tool Calling Router]
  tool-calling-router --> skill-playbooks[Skill Playbooks]
  skill-playbooks --> arize-platform-data[Arize Platform Data]
  alex-copilot --> arize-platform-data
  arize-platform-data --> evaluation-framework[Evaluation Framework]
  evaluation-framework --> cross-functional-reviewers[Cross Functional Reviewers]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`User`	Platform user asking for debugging, monitoring, prompt optimization, or evaluation help.	`Alex Copilot`
`Alex Copilot`	LLM assistant embedded in Arize's observability platform.	`User`, `Tool Calling Router`, `Skill Playbooks`, `Arize Platform Data`, `Evaluation Framework`
`Tool Calling Router`	OpenAI tool-calling layer that selects top-level and nested skills.	`Alex Copilot`, `Skill Playbooks`
`Skill Playbooks`	Structured workflows derived from solutions-architect debugging and optimization playbooks.	`Tool Calling Router`, `Arize Platform Data`
`Arize Platform Data`	Traces, sessions, evals, monitors, prompts, and platform artifacts used by Alex.	`Alex Copilot`, `Evaluation Framework`
`Evaluation Framework`	Layered evaluations for tool-call correctness, task correctness, trace order, and session outcomes.	`Arize Platform Data`, `Cross Functional Reviewers`
`Cross Functional Reviewers`	Founders, solutions architects, engineers, and product stakeholders defining requirements and dogfooding Alex.	`Evaluation Framework`

Data flow

A user asks Alex to debug or optimize an ML/LLM workflow; the copilot selects a skill such as model-performance debugging, prompt optimization, monitor setup, or eval-template generation; traces and tool calls are captured in Arize; and evaluation checks the tool, task, trace, and session levels.

“and you give the LLM maybe four tools that it can call, right? One is debug the performance of my model. Another one could be what S was saying, hey, answer a generic like documentation question.”transcript, stated

“the top level but then each individual step that's taking each of those spans really has an evaluation. Did it call the right it call the tool with the right arguments? Did it accomplish the task for that tool correctly? I'm trying to think was the like for AI search was”transcript, stated

Why this architecture

The initial decision-tree architecture fit the model capability and known expert playbooks at the time, while later model improvements justified making Alex less on-rails.

“we put it on rails so we used open AI tool calling a lot and the architecture of our co-pilot was really like a decision tree”transcript, stated

“I think what we're doing today is we're starting to rethink the way our Alex works and make it less on rails like what I was talking about and give it a little bit more autonomy, a little bit more generic tools that it can try to compose. So we're in this like”transcript, stated

Models used: GPT-3.5 · OpenAI tool calling · LLMs

Scale

Development started around November 2023 with a July 2024 conference launch deadline.
“So, I don't know if he mentioned, but we started in like November 2023 and then we had like a deadline. We knew in July 2024 we were going to have our conference and we were going to release this. And so Jack and I were on a timeline like we knew we had to get this”transcript, stated
The team dogfooded weekly with new skills and changes.
“thing in shape. So we just would do weekly dog fooding with new skills or changes and we would just be like try it. And then once we started getting into the UI, it was like behind a feature flag so they could demo it or they could try it out in their own time where they we would do a lot of oh”transcript, stated
A trace-level evaluation may inspect hundreds of spans.
“you have a trace with hundred spans like that's very overwhelming. You don't need to think about it. it was like, "Oh, I need to have an eval at every one of those those points." What you do need is”transcript, stated

Alternatives considered (roads not taken)

A single prompt released as a feature.

Rejected because: The team found AI products require substantial product, workflow, eval, dogfooding, and platform work beyond prompt writing.

“there's so many teams out there that just still frame AI products as I'm going to write a prompt and we're going to release it as a feature”transcript, stated

Fully autonomous agent from the start.

Rejected because: The first version encoded known solutions-architect playbooks into an on-rails workflow better suited to the model and use case at the time.

“it. It's really an on rails like workflow experience. And that worked well for what we're trying to do, which is our essays and ces. They've seen enough customer models where they really do have maybe it's in their head, maybe it's written down, but they have a”transcript, stated

Generic off-the-shelf evals only.

Rejected because: Custom platform and task errors require application-specific evaluations at multiple levels.

“>> you are going to have custom errors that an off-the-shelf eval is never going to find for you. >> Totally. And it's like that it's that intuition building I told you about like early on like that's the exact key there”transcript, stated

Patterns

`tool-use-orchestration` (agent-runtime)

Alex used OpenAI tool calling to select among debugging, documentation, monitor, prompt, and eval skills.

“we used open AI tool calling a lot”transcript, stated

`eval-driven-development` (evaluation)

Arize evaluates at tool-call, task, trace, and session levels instead of relying on one aggregate result.

“you're going to evaluate each different level.”transcript, stated

“did it call all the right tools in the right order and then assession like I care is my user getting frustrated”transcript, stated

`observability-tracing` (debugging)

The team used traces and its own observability product to understand and debug Alex behavior.

“anybody who's started to build an AI product and like really looks at their data and is looking at their traces and seeing what's happened”transcript, stated

Tradeoffs

Initial autonomy level

Chose On-rails decision tree with tool calling over Open-ended autonomous reasoning.

Rationale: Known solutions-architect playbooks were reliable enough to encode into structured skills.

“they have a playbook of, hey, customer looks like they're experiencing this type of issue.”transcript, stated

Evaluation design

Chose Layered custom evals over Off-the-shelf evals only.

Rationale: Real product errors appear in tool arguments, spans, traces, and user frustration, not just final text.

“Did it call the right it call the tool with the right arguments? Did it accomplish the task for that tool correctly?”transcript, stated

Gotchas (what bit them)

AI product work is much more than prompt writing.

Root cause: Production assistants need UI, skills, evals, dogfooding, traces, and cross-functional requirements.

Resolution: Run weekly dogfooding, involve solutions architects and engineers, and evaluate multiple levels.

“all the work that we know goes into it that is not just you're writing a prompt.”transcript, stated

One generic eval cannot judge every agent task.

Root cause: Different skills have different correctness requirements.

Resolution: Convert stakeholder requirements into per-task evaluation templates.

“task correctness cannot be one eval. It's different my tasks.”transcript, stated

Evaluation

Alex was tested through weekly dogfooding, feature flags, stakeholder requirements, Arize traces, and custom eval templates for QA correctness, tool calls, task completion, trace order, and session outcomes.

Metrics: Tool-call correctness · Tool argument correctness · Task success · Trace tool order · Session-level user frustration/memory

“we just would do weekly dog fooding with new skills or changes”transcript, stated

“we have things at like QA correctness at the top level but then each individual step”transcript, stated

Outcomes

Alex launched from a November 2023 start toward a July 2024 conference release.
The initial on-rails design captured expert support playbooks in LLM tool workflows.
Arize established a layered evaluation process for agentic observability workflows.

The source notes an eight-month build window, weekly dogfooding, and evaluation across tool, task, trace, and session levels.

“started in November and you had a Jul July deadline”transcript, stated