Building Alex: An Agent-First AI Engineering Assistant with Production-Grade LLMOps

alyx · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI engineering assistant inside an observability platform

Arize needed Alex, an AI engineering agent, to stay on task across complex workflows, manage very large trace/eval/playground data without context overflow, test non-deterministic behavior in CI, and debug production agent failures using observability data.

Constraints:

• Agent tasks span traces, evals, sessions, playground interactions, tool calls, skills, and memory.
• Plans can be buried or truncated if stored only in conversation history.
• Trace and eval data can be too large for the context window.
• Production agent behavior is non-deterministic and cannot be validated with vibe checks.

Architecture

Alex uses a structured planning toolset with persisted to-do state, an in-progress pointer, finish-gate errors, and plan reinjection on every LLM call. For large data, it stores full JSON server-side, gives the model previews and IDs, and exposes jq/grep-style tools with hard output budgets. For quality, it turns production traces into golden examples, runs decision-point and trajectory tests in CI, uses LLM judges, and exposes traces/evals/feedback as agent skills for debugging.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user[User] --> alex-agent[Alex Agent]
  alex-agent --> planning-tools[Planning Tools]
  planning-tools --> plan-store[Plan Store]
  planning-tools --> finish-gate[Finish Gate]
  plan-store --> alex-agent
  alex-agent --> large-json-store[Large JSON Store]
  large-json-store --> composable-data-tools[Composable Data Tools]
  composable-data-tools --> alex-agent
  alex-agent --> production-trace-corpus[Production Trace Corpus]
  production-trace-corpus --> ci-evaluation-harness[CI Evaluation Harness]
  ci-evaluation-harness --> llm-judge[LLM Judge]
  production-trace-corpus --> observability-skills[Observability Skills]
  observability-skills --> alex-agent

Components

Data flow

A user asks Alex to inspect traces, evals, or playground behavior; Alex writes and updates a persisted plan, injects that plan into every LLM call, uses tools and large-JSON references to query data incrementally, records production traces, replays golden traces in CI, and gives the agent observability skills so it can investigate runtime failures.

Why this architecture

The architecture treats planning, data access, tests, and debugging as first-class runtime systems because prompts alone could not keep the agent focused, context-safe, testable, or able to inspect production behavior.

Models used: LLMs · LLM judge · Claude Code Review

Scale

• Large JSON truncates every value to 100 characters max in previews.
  “it truncates every single value inside of a cell to 100 characters max, which means that every field, every key, every array element um that appears in that preview, you know, except for like really long”transcript, stated
• No tool-call output can exceed 10,000 characters.
  “kind of hard budget, something like no tool call output can exceed 10,000 characters. Which meant that no matter how large the actual underlying data was, each query actually cost a predictable amount of context.”transcript, stated
• CI experiments run live for every agent change, including prompts and tool calls.
  “Anytime anyone now makes a change to our agent, there's experiments in CI that run live. Anything from a prompt to a tool call.”transcript, stated
• Alex's context includes traces, evals, sessions, skills, planning, memory, and tool calls.
  “has context about all of your traces. It has context about your evals. It understands your sessions. And it also it has a whole harness underneath it that does planning, has access to tool calls and skills, and then memory that's used across the sessions that people do”transcript, stated

Alternatives considered (roads not taken)

Store the plan inside conversation history.

Rejected because: The plan conflicts with user messages and can get buried or truncated.

Dump large trace/eval JSON into the context window.

Rejected because: That causes context overflow and hides the data structure the agent needs for useful queries.

Hand-write expected inputs for tests.

Rejected because: Production traces better represent what good behavior actually looks like.

Patterns

agent-memory (planning)

Alex stores plans outside the conversation and reinjects state into every LLM call so the agent has a persistent working pointer.

context-compaction (context)

Large JSON stores full data server-side, gives the model a structural preview, and uses tools to retrieve only needed slices.

tool-use-orchestration (agent-tools)

Alex exposes jq, grep JSON, planning, trace, eval, and observability tools as composable skills.

llm-as-judge (evaluation)

Trajectory-level tests replay production traces and score whole-agent behavior with an LLM judge.

observability-tracing (debugging)

Alex uses production traces, evals, feedback, CLIs, and observability telemetry as debugging inputs.

Tradeoffs

Plan storage

Chose Plan outside conversation history, reinjected every call over Plan embedded in the chat transcript.

Rationale: The plan stays visible and is not buried by user messages or tool history.

Large data access

Chose Reference-plus-preview with jq/grep tools over Full dataset in context.

Rationale: The agent can query unlimited underlying data while each tool call uses predictable context.

Test data source

Chose Production traces as golden examples over Hand-written expected inputs.

Rationale: Real sessions capture tool order, LLM behavior, and user intent better than synthetic test fixtures.

Gotchas (what bit them)

Agents without an in-progress state lose focus mid-task.

Root cause: Pending/completed states do not tell the agent what it is doing right now.

Resolution: Add an in-progress state and reinject the current plan before noisy tool history.

Previewing only the first rows of large JSON hides schema structure.

Root cause: Rows can be token-heavy and may not represent the full JSON shape.

Resolution: Compress values but preserve the full structure so the agent can query correctly.

Unit tests and stubs can miss production tool-call failures.

Root cause: Renaming a tool call produced a runtime failure invisible to stubbed/unit tests.

Resolution: Replay production traces through the real orchestrator in CI.

Coding agents cannot debug non-deterministic agents from code alone.

Root cause: The important behavior lives in traces because each agent decision can vary at runtime.

Resolution: Give the agent trace, eval, feedback, and telemetry access through skills/CLI.

Evaluation

Alex uses production traces as ground truth, decision-point pytest checks for specific agent decisions, trajectory-level tests scored by LLM judges, and live CI experiments on prompt/tool changes.

Metrics: Task completion rate · Decision-point correctness · Trajectory success · Tool-call runtime failures · LLM judge score

Outcomes

• In-progress plan state drastically improved task completion rates.
• Large JSON enabled the agent to manage effectively unlimited context through previews and tools.
• Production trace replay and LLM-judge trajectory tests caught tool-call failures in CI.
• Observability skills let the agent use traces, evals, and feedback for debugging.

Alex uses 100-character value truncation in large-JSON previews, 10,000-character tool-output budgets, production traces as CI replay inputs, and live CI experiments for prompt and tool changes.