Building Reliable AI DevOps Agents: Engineering Practices for Nondeterministic LLM Output

trunk · Tech · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI DevOps agents for CI failure analysis

Trunk wanted an AI agent to perform DevOps and developer-experience work in CI, especially root-cause analysis for test failures, while controlling nondeterministic LLM output enough for production use.

Constraints:

• LLM outputs are inconsistent, can be overly verbose, and may present different answers for the same inputs.
• A broad everything-agent scope makes it hard to test failure modes or know whether the agent is useful.
• Tool calling must be reproducible enough for CI-oriented workflows.
• Large CI logs and historical failure data can exceed context limits unless preprocessed.
• User-facing PR comments must be actionable and concise, not long unfocused essays.
• The non-LLM system, tool executions, structured outputs, and failure handling still need conventional tests.

Architecture

Trunk built a narrow RCA agent for CI test failures instead of an everything agent. It uses existing Flaky Tests infrastructure and historical stack traces, preprocesses and groups similar failures to fit the LLM context window, and posts summaries to GitHub PRs. The team changed from Claude to Gemini when Gemini produced more deterministic tool calls, accepted some reasoning tradeoff, and wrapped the model with system prompts, tool execution, structured output validation, capped reruns, subagents for relevant structured extraction and summarization, LangSmith observability, unit tests with mocked LLM responses, integration/e2e tests that call the LLM, and PR-comment feedback forms for internal dogfooding.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  ci-failure-inputs[CI failure inputs] --> flaky-tests-history[Flaky Tests history]
  ci-failure-inputs --> failure-preprocessor[Failure preprocessor]
  flaky-tests-history --> rca-agent[RCA agent]
  failure-preprocessor --> rca-agent
  rca-agent --> gemini-model-layer[Gemini model layer]
  gemini-model-layer --> tool-execution-layer[Tool execution layer]
  tool-execution-layer --> langsmith-observability[LangSmith observability]
  rca-agent --> structured-output-validator[Structured output validator]
  structured-output-validator --> subagent-summarizers[Subagent summarizers]
  subagent-summarizers --> github-pr-comment[GitHub PR comment]
  structured-output-validator --> github-pr-comment
  github-pr-comment --> feedback-form[Feedback form]
  feedback-form --> langsmith-observability
  langsmith-observability --> test-suite[Test suite]
  test-suite --> rca-agent

Components

Data flow

A CI failure and its historical stack traces flow from Trunk's Flaky Tests infrastructure into preprocessing that groups similar failures and removes irrelevant log data. The RCA agent analyzes the reduced failure context with a model layer selected for reliable tool calls, uses tools whose executions are monitored, validates structured and user-facing output, reruns bounded failures when cost-effective, posts concise summaries to GitHub PRs, and collects developer feedback from the same PR comments. Unit tests mock LLM responses, while integration/e2e tests and LangSmith traces expose real LLM behavior and regressions across the whole workflow.

Why this architecture

Trunk chose a narrow RCA workflow because small, real-data use cases make failure modes visible quickly and let the team pivot before building an overbroad agent. It switched models because prompt pressure alone did not make tool calling deterministic enough. It tests the non-LLM system with mocks while still running integration and end-to-end LLM calls because both conventional software behavior and live model weirdness matter. It validates output and uses subagents because developer-facing PR comments must be concise and actionable.

Models used: Claude for early prompt/tool-calling experiments · Gemini for more reproducible tool calling · LLM network responses mocked with Vercel AI SDK tooling · Subagents for structured extraction and summarization

Scale

• The RCA agent started with one focused use case: root-cause analysis for test failures.
  “At Trunk, we started by focusing on root cause analysis (RCA) for test failures.”article, stated
• Trunk's explicit quality goal was to avoid an agent that fails about half the time.
  “build an agent that doesn’t suck 50% of the time.”article, stated
• The recommended initial scope is one or two use cases rather than a broad everything agent.
  “Pick out one or two use cases, and get them working well.”article, stated
• The agent is dogfooded on Trunk's main monorepo so the whole team sees real outputs.
  “We use our agent on our main trunk monorepo so everyone is exposed to what we’re building.”article, stated

Alternatives considered (roads not taken)

Build an everything agent

Rejected because: A mega-prompt and many tools do not guarantee the agent will use tools properly, and broad scope delays learning from real failures.

Keep trying to prompt Claude into deterministic tool calls

Rejected because: After extensive prompt tweaking, switching to Gemini made tool calls more reproducible.

Rely only on LLM-calling integration tests

Rejected because: The surrounding tools, inputs, outputs, and structured-response handling still need deterministic unit tests with mocked LLM responses.

Feed raw CI logs and all history directly into the LLM

Rejected because: A single CI log can blow the context window, so grouping and removing irrelevant data is necessary.

Let the agent write long PR-comment essays

Rejected because: Even correct information fails as user experience when developers cannot quickly find the actionable part.

“typewriters will eventually write the complete works of Shakespeare. And at any given time, an LLM might write Hamlet in a PR comment. They occasionally output like they are in high school, desperately trying to reach the assignment word limit. Good user”article, stated

“correct information that helps solve a problem, but it is a failed experience for users if the info is buried in a novel. But unlike traditional”article, stated

Patterns

eval-driven-development (quality)

Trunk combines mocked unit tests, integration tests, evals, and end-to-end workflow tests so prompt and agent changes can be measured rather than guessed.

model-routing (modeling)

The team changed the agent's model when empirical tool-calling reliability mattered more than Claude's reasoning quality for this workflow.

observability-tracing (monitoring)

LangSmith tracks the agent's inputs, outputs, and tool calls, while feedback forms expose which PR-comment results developers found useful or problematic.

context-compaction (context)

The system groups similar failures and removes irrelevant CI-log information to keep historical test data within the model context window.

structured-output-contract (reliability)

Trunk validates LLM inputs and outputs and reruns when deterministic output constraints, such as character limits, fail.

tool-use-orchestration (architecture)

The RCA agent uses tools and subagents for structured extraction and summarization, with tool executions monitored as potential failure points.

human-in-the-loop-gate (feedback)

Internal developers and project managers dogfood the agent and provide feedback before broader user exposure.

Tradeoffs

Start with RCA for test failures

Chose A narrow CI failure-analysis agent over A general DevOps everything agent.

Rationale: A focused use case used existing failure data, exposed edge cases quickly, and produced something real users could try.

Switch from Claude to Gemini

Chose More reproducible Gemini tool calls over Claude plus increasingly forceful prompt tuning.

Rationale: Tool-call determinism mattered enough that Trunk accepted some reduction in reasoning quality.

Mock LLM responses in unit tests

Chose Deterministic service-style unit tests over Calling the live LLM for every test case.

Rationale: Mocking enables repeatable tests for tools, inputs, outputs, and error handling while live tests remain useful for discovering bad model behavior.

Preprocess failure logs

Chose Grouped and trimmed failure context over Passing full CI logs and all history directly to the model.

Rationale: Preprocessing preserves relevant signal while avoiding context-window blowups from large logs.

Use bounded reruns for deterministic output failures

Chose Rerun on validation failure when cost-effective and capped over Accepting malformed or overlong user-facing output.

Rationale: Reruns can improve output quality when prices allow, but the retry budget must be capped.

Gotchas (what bit them)

Prompt emphasis could not make Claude's tool calls deterministic enough.

Root cause: The team kept escalating prompt wording, but the model still did not call tools reliably for the workflow.

Resolution: Switch the model powering the agent to Gemini and accept a reasoning-quality tradeoff.

An everything-agent scope hides whether the agent can solve any task well.

Root cause: Adding many tools and a mega-prompt does not teach the model how to use them properly.

Resolution: Start with one or two use cases and test against real data quickly.

Raw CI logs can blow the model context window.

Root cause: A single log may be too large, and historical runs multiply that context pressure.

Resolution: Group similar failures and strip non-relevant information before the LLM sees the data.

Correct RCA content can still fail the user if it is too verbose.

Root cause: The model may bury actionable information in long, essay-like PR comments.

Resolution: Treat output format as user experience, validate deterministic output constraints, and use subagents to extract and summarize.

Testing only the LLM output misses failures in the rest of the system.

Root cause: Tools, input packaging, structured output handling, and non-LLM code paths can break independently of the model response.

Resolution: Use traditional unit tests with mocked LLM responses, plus integration and e2e tests for real model behavior.

Evaluation

Trunk evaluates reliability with layered tests and real usage feedback: unit tests mock LLM network responses, integration and e2e tests call the LLM for full workflow behavior, evals and observability catch regressions and prompt A/B effects, LangSmith traces inputs/outputs/tool calls, and developer feedback forms on PR comments expose good and bad outputs from dogfooding.

Metrics: Deterministic tool-call reproducibility · Structured input and output validation pass/fail · Character-limit and verbosity checks · Unit, integration, eval, and end-to-end workflow test results · Tool execution errors and LLM output errors · Positive and negative feedback from PR-comment forms · Regression and A/B prompt-tweak outcomes

Outcomes

• Trunk produced an RCA agent for flaky tests and CI failures rather than a broad everything agent.
• Switching to Gemini made tool calls more reproducible, with a stated reasoning-quality tradeoff.
• The system gained more reliable tests and feedback loops for incremental changes.
• LangSmith traces and PR-comment feedback forms give the team visibility into both good and bad outputs.
• The article does not report quantitative production lift, adoption, latency, or cost metrics.

The source reports a focused RCA use case, one or two use cases as the recommended initial scope, a target of avoiding an agent that fails 50% of the time, dogfooding on the main Trunk monorepo, and a waitlist for a CI-failure RCA agent. It does not disclose user counts, CI volume, latency, or cost.