Building Production-Ready AI Agents and Monitoring Systems

portkey,_airbyte,_comet · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production LLM and agent infrastructure across AI gateways, observability, data access, and durable workflows

Practitioners from Portkey, Airbyte, Comet, Hasura, DBOS, and Toolhouse described the infrastructure required to make LLM applications and agents reliable in production: observability, cost control, forward-compatible model interfaces, guardrails, data-quality loops, durable execution, human approval gates, and deterministic data access.

Constraints:

• LLM applications change underneath teams as model APIs, costs, and capabilities evolve.
• LLM logs and embedding payloads are much larger than traditional application logs, so naive logging can create unexpected cost and storage problems.
• Failures are not simple HTTP-style success/failure events; hallucination, bias, poor routing, and weak answers require qualitative monitoring and guardrails.
• Agent workflows can span many steps, external APIs, human approvals, and asynchronous callbacks, so crashes can duplicate writes or lose state.
• Enterprise agents need to move large data sets without pushing everything through an LLM context window.
• Production teams need evaluation suites before deployment because prompt and model changes otherwise overfit to developer intuition.

Architecture

The panel describes a production LLMOps stack rather than a single product. Applications sit behind gateway and model-interface layers that provide model routing, guardrails, fallbacks, and forward compatibility. Observability systems record traces, spans, inputs, outputs, costs, and custom metrics such as hallucination or relevance. Data-access systems keep retrieval and bulk enterprise data movement deterministic instead of stuffing 100,000 rows into model context. Agent runtimes add tool/function calling through MCP or tool catalogs, with durable workflow engines persisting step results so long-running agents can wait for callbacks or human approval and then resume. Evaluation suites, holdout sets, and PR-level feedback loops close the development loop before production deployment.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  production-llm-application[Production LLM application] --> ai-gateway[AI gateway]
  production-llm-application --> observability-and-evaluation-layer[Observability and evaluation layer]
  production-llm-application --> unified-data-access-layer[Unified data access layer]
  production-llm-application --> durable-workflow-runtime[Durable workflow runtime]
  ai-gateway --> llm-provider-interface[LLM provider interface]
  ai-gateway --> guardrail-policies[Guardrail policies]
  ai-gateway --> mcp-and-tool-servers[MCP and tool servers]
  ai-gateway --> observability-and-evaluation-layer
  llm-provider-interface --> evaluation-suite[Evaluation suite]
  guardrail-policies --> human-approval-gate[Human approval gate]
  observability-and-evaluation-layer --> evaluation-suite
  unified-data-access-layer --> enterprise-data-sources[Enterprise data sources]
  unified-data-access-layer --> structured-memory[Structured memory]
  enterprise-data-sources --> connector-builder[Connector builder]
  connector-builder --> production-llm-application
  durable-workflow-runtime --> human-approval-gate
  durable-workflow-runtime --> external-services[External services]
  mcp-and-tool-servers --> external-services

Components

Data flow

A production LLM or agent request enters an application, then typically passes through a pluggable model or AI gateway layer. The gateway can call LLMs, apply guardrails, route to fallback models, redact unsafe output, and expose tools through MCP or function-calling infrastructure. Application traces flow to observability systems that store inputs, outputs, spans, metrics, and costs for debugging and evaluation. Data-heavy agents avoid passing all records through model context by asking a unified data access layer for deterministic retrieval from APIs, databases, RAG, text-to-SQL, or semantic search. Long-running agents are wrapped in durable workflow steps that persist outputs, wait for callbacks or human approval, and resume after crashes. Evaluation suites and holdout datasets compare prompt/model changes before release.

Why this architecture

The architecture centralizes cross-cutting production concerns instead of making every application team rebuild observability, retries, cost controls, guardrails, data movement, and human approval from scratch. A gateway absorbs model churn and qualitative failure handling; tracing and evals make behavior debuggable; durable execution protects long-running agent state; and unified data access prevents enterprise data from being squeezed through fragile LLM context.

Models used: LLMs · OpenAI models · Claude · Perplexity · Gemini · GPT-3.5 Turbo · Vision models · Smaller task-specific models · Embedding models · RAG systems · OpenAI Swarm · MCP · DBOS · Portkey AI Gateway · Comet Opik · PromptQL

Scale

• Airbyte had built roughly 400 source connectors and users had built roughly 10,000 connector-builder configurations.
  “we do connectors uh to apis and we've built 4 100 or so Source connectors for various apis”transcript, stated

  “people have built 10,000 of those because there's bunch of crazy stuff out there”transcript, stated
• The connector builder aims to turn API documentation into a data connector in seconds or minutes.
  “go from from nothing to a um you know a fully fledged data connector in seconds or minutes”transcript, stated
• Enterprise data-transfer examples include moving 100,000 rows or seven weeks of customer support tickets between agent tasks.
  “what if I want to transfer a 100,000 rows of data”transcript, stated

  “my last 7 weeks customer support tickets another one another agent which specifically does let's say CH churn risk analysis”transcript, stated
• Long-running agent workflows may wait days or weeks for human input.
  “the customer service agent may need to wait for some user input and that weight will last for like a week for example”transcript, stated

  “we provide Primitives to help you wait for say a few days or a few weeks and then you can continue from where you left out”transcript, stated
• The MCP demo exposed 88 servers and hundreds of tools through the gateway.
  “I have access to 88 servers that are available today”transcript, stated

  “lots of possibilities are around what you could do with 88 different servers and um you know hundreds of tools”transcript, stated
• The Opik demo traces a RAG bot as a trace with three spans and metrics such as hallucination and answer relevance.
  “you've got inputs and outputs and you've got these hallucination and answer relevance metrics”transcript, stated

  “you can see these different spans there's three here in this Trace”transcript, stated
• Cost discussion included concrete per-interaction examples such as four cents for a hover interaction and expensive embedding logs.
  “when they hover over this it cost us 4 cents is what you're telling me”transcript, stated

  “or every time we log a embedding it's $9”transcript, stated
• The Toolhouse example described a daily X monitoring agent costing about twelve cents.
  “we don't have to pay for any like sentiment analysis or like tweet monitoring solution just to hour every day 12 cents”transcript, stated

Alternatives considered (roads not taken)

Naive CloudWatch or traditional logging for LLM traces

Rejected because: LLM logs and embeddings are much larger than ordinary logs, so traditional logging infrastructure can create runaway costs.

All-or-nothing generation for API connectors

Rejected because: Many generation steps compound failure probability, so Airbyte preferred a co-pilot workflow with previews and user guidance.

Choosing smarter models first when output quality disappoints

Rejected because: The Airbyte speaker argued that poor inputs and data organization are usually the real problem.

Putting all enterprise data transfer through LLM context

Rejected because: Large tables and support-ticket corpora lose determinism and trigger hallucination/context-loss risks when moved through model context.

AWS Step Functions and Lambda reference architecture for async agent callbacks

Rejected because: DBOS positioned that approach as operationally complex for agents that need callbacks, retries, and human approvals.

Manual snapshot and restore for human-in-the-loop agents

Rejected because: The panel described existing frameworks as requiring explicit state capture and restore, while durable execution persists step results and resumes.

Hardcoded per-task agent trees for every business query

Rejected because: PromptQL's speaker argued that customer success users can ask many combinations of retrieval, analysis, and action, so a programmatic runtime and unified data access layer are more scalable.

Patterns

observability-tracing (production-operations)

Comet/Opik instruments LLM applications with traces, spans, inputs, outputs, and task-level metrics so teams can debug the exact step where a RAG or agent flow breaks.

guardrails-io-filtering (safety-and-governance)

Portkey's gateway centralizes guardrail checks such as bias detection, redaction, and fallback routing instead of forcing every application to implement those checks itself.

model-routing (model-operations)

The panelists recommended pluggable model interfaces and gateway-level routing so applications can swap providers, use newer APIs, or choose more capable models after guardrail failures.

durable-execution (agent-runtime)

DBOS wraps agent steps in durable workflows that persist outputs, wait for callbacks or human approvals, and resume from the last completed step after crashes.

human-in-the-loop-gate (safety-and-governance)

Agents flag irreversible actions such as refunds, credits, database writes, and POST API calls for human approval before execution.

tool-use-orchestration (agent-tools)

MCP servers and Toolhouse-style function platforms expose external tools to agents so the LLM can select tools, call them, receive results, and continue a loop.

eval-driven-development (quality)

Teams compare prompts, models, and changes against holdout data and automated feedback suites rather than eyeballing a few examples.

cost-aware-routing (production-operations)

The panel recommends tracking cost per LLM call, limiting retry loops, keeping cheaper models available for substeps, and making routing levers tunable.

data-quality-loop (data-operations)

Airbyte frames many LLM failures as poor data-input problems, pushing teams toward better summaries, metadata, chunk organization, and deterministic data access instead of only upgrading models.

structured-output-contract (agent-runtime)

PromptQL constrains agent data movement through programmatic runtime artifacts and structured memory, letting follow-up steps reference artifacts instead of reserializing records into chat context.

Tradeoffs

Use gateway-level guardrails and routing

Chose Central AI gateway between applications and LLMs over Embedding every guardrail, fallback, and redaction path inside each application.

Rationale: Centralizing the policy and routing layer lets application teams focus on product workflows while still handling qualitative LLM failures and provider changes.

Build connectors with a co-pilot workflow

Chose Guided connector generation with previews over One-shot, all-or-nothing connector generation from documentation.

Rationale: The guided path reduces compounded hallucination/error risk while still accelerating connector creation from docs.

Prefer better data preparation before model escalation

Chose Summaries, metadata, chunk organization, and data quality work over Immediately switching to a smarter or more expensive model.

Rationale: The panel's Airbyte perspective is that many unsatisfactory outputs come from bad inputs rather than insufficient model capability.

Make models pluggable and routing tunable

Chose Swappable LLM interfaces plus cheaper models for selected steps over Hardwiring a single frontier model throughout every workflow.

Rationale: Provider churn, cost pressure, and heterogeneous substeps require fast model substitution and cost controls without rewriting the product.

Use durable execution as a library

Chose Decorated normal Python/TypeScript functions with persisted step outputs over Manually building DAGs, Step Functions, queues, and callback plumbing.

Rationale: The library approach lets teams write natural workflows while still supporting retries, callbacks, human waits, and crash recovery.

Move data orchestration into programmatic runtime

Chose Unified data access layer plus structured memory artifacts over Multiple specialized agents passing raw enterprise data through LLM context.

Rationale: The programmatic/data-access approach keeps large data movement deterministic and lets follow-up logic operate over saved artifacts instead of fragile chat memory.

Gotchas (what bit them)

Naively self-hosting embeddings can fail for ordinary operations such as health checks.

Root cause: Embedding long documents consumed enough memory and time that service health checks killed nodes, creating unexplained failures.

Resolution: Treat embeddings as production infrastructure, size memory and timeouts deliberately, and consider managed embedding services when the operational burden outweighs savings.

Production AI systems often ship without enough monitoring.

Root cause: Some AI practitioners come from math, analytics, or data science backgrounds where software-style logging and monitoring are not yet common knowledge.

Resolution: Define failure modes, logs, metrics, and dashboards before deployment, then use traces and spans for retroactive debugging.

Traditional log stores can make LLM observability unexpectedly expensive.

Root cause: LLM traces, prompts, outputs, and embeddings can be far larger than normal application logs.

Resolution: Use LLM-aware tracing and cost metrics, avoid logging embeddings blindly, and choose storage systems appropriate for large token logs.

All-or-nothing generation lets small error probabilities multiply into product failure.

Root cause: A chain of multiple generated connector decisions each carries its own failure chance, so unattended generation compounds errors.

Resolution: Use a co-pilot workflow with previews and user guidance at each step instead of one-shot connector generation.

Agent-to-agent data transfer through LLM context causes hallucination and context loss.

Root cause: Enterprise workflows need to pass bulk structured and unstructured data, but LLM context is a poor transport layer for large data sets.

Resolution: Move orchestration into a programmatic runtime and delegate data retrieval to a deterministic unified access layer with structured memory artifacts.

Long-running agents can duplicate writes or lose work after crashes.

Root cause: Agents may wait days for callbacks or human input, but restarting from the beginning repeats side effects and loses context.

Resolution: Persist every step result and resume from the last completed step, with explicit waiting primitives for callbacks and human approval.

Human-in-the-loop review is easy to miss if mutation detection is probabilistic.

Root cause: Agents can call tools that write databases, post APIs, issue credits, or process refunds, and missing one irreversible action bypasses human control.

Resolution: Deterministically flag mutation-capable tools and route those steps through explicit approval gates.

Evaluation

The panel recommends treating LLM systems as software systems with pre-production eval suites, holdout datasets, prompt/model version comparisons, PR-level red/green feedback, custom production metrics, tracing, and cost monitoring. For agent systems, evaluation also includes observing end-state behavior after tool calls, retries, callbacks, and human approval gates.

Metrics: Trace inputs and outputs · Span-level retrieval and LLM-call logs · Hallucination metrics · Answer relevance metrics · Bias/fairness custom metrics · Prompt-version comparison on holdout data · PR-level eval regressions and improvements · Unit, integration, and end-to-end style AI tests · Cost per LLM call and workflow step · Retry loop counts and workflow step counts

Outcomes

• Airbyte's documentation-driven connector builder can create data connectors in seconds or minutes, with the co-pilot design mitigating all-or-nothing generation failures.
• Portkey positions the AI Gateway as a reusable layer for guardrails, governance, routing, and forward compatibility in production generative AI applications.
• Comet's Opik demonstrates open-source LLM tracing with spans, inputs, outputs, hallucination metrics, answer relevance, and custom metrics.
• DBOS demonstrates a refund agent that updates a database, sends approval email, waits asynchronously, and resumes after human input.
• PromptQL frames unified data access and structured memory as a way to answer complex customer-success questions across tickets, billing, metrics, and support conversations.
• Toolhouse demonstrates agents selecting tools and producing small business workflows such as newsletters, sentiment digests, and gift recommendations.

The panel includes concrete production-scale claims rather than one deployment benchmark: Airbyte had about 400 source connectors and 10,000 connector-builder configurations; agent data-transfer examples include 100,000 rows and seven weeks of support tickets; DBOS workflows can wait days or weeks; Portkey's MCP demo used 88 servers and hundreds of tools; and Opik traced RAG calls with spans and hallucination/relevance metrics.