Lessons Learned from Production AI Agent Deployments

google_/_vertex_ai · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: production AI agents

Google's Vertex Applied AI Incubator observed that production agents fail when teams treat the model as the system and omit prompt optimization, layered safety, versioning, and stage-level evaluations.

Constraints:

• Production agents include orchestration, grounding, tools, APIs, CI/CD, analytics, safety, and evaluation around the model.
• Public agents face prompt injection, bad actors, long-turn attacks, API errors, and output safety problems.
• End-to-end evaluation can show that quality fell but cannot localize whether query rewrite, retrieval, reranking, or summarization caused the loss.
• Agent prompts, tools, and functions need version control like normal software artifacts.

Architecture

The talk recommends a production-agent architecture with meta-prompting for high-fidelity prompts, iterative prompt optimization using evaluation feedback, layered input/agent/output safety controls, semantic caching where repeated queries do not need the agent, golden-dataset evaluations across each pipeline stage, and Git-based versioning for prompts, functions, and tools.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Meta_prompting_system["Meta-prompting system"] --> Target_agent_system["Target agent system"]
  Target_agent_system --> Safety_filters["Safety filters"]
  Target_agent_system --> Semantic_cache["Semantic cache"]
  Target_agent_system --> Evaluation_harness["Evaluation harness"]
  Evaluation_harness --> Meta_prompting_system
  Evaluation_harness --> Analytics_feedback["Analytics feedback"]
  Analytics_feedback --> Safety_filters
  Target_agent_system --> Version_controlled_artifacts["Version-controlled artifacts"]

Components

Data flow

A developer seeds a meta-prompting system with task context, which generates or optimizes prompts for a target agent. The target agent handles user traffic through safety filters, APIs, grounding, and generation controls, while cached responses can bypass LLM calls for repeated queries. Runtime responses are compared with expectations in an evaluation harness, and stage-level metrics identify whether failures come from query rewriting, retrieval, reranking, summarization, or tool calling. Agent artifacts are stored in Git so changes can be diffed and rolled back.

Why this architecture

The talk argues that models will commoditize, so production quality comes from the ecosystem around the model: prompt quality, safety layers, caching, evaluations, and software lifecycle discipline. Stage-level evaluation is needed because top-level response quality alone does not reveal which part of an agent or RAG pipeline regressed.

Models used: Gemini · OpenAI models · Claude · Vertex Prompt Optimizer · DSPy · AdalFlow · Dialogflow CX / Conversational Agents

Scale

• The speaker says the lessons come from delivering hundreds of models into production with developers, customers, and partners.
  “after delivering hundreds of models into production with lots of different developers and customers and partners”transcript, stated
• The speaker has more than 12 years of conversational AI and NLP experience.
  “I have been in the conversational Ai and NLP space for a little over 12 years”transcript, stated
• Session limits of 30 to 50 turns are presented as a safety control against long-tail prompt injection patterns.
  “not allowing your users to talk over 30 turns or 50 turns or something like that”transcript, stated

Alternatives considered (roads not taken)

Focus mainly on model choice

Rejected because: The speaker says production agents are ecosystems with grounding, prompt engineering, orchestration, APIs, CI/CD, and analytics.

Rely on prompt engineering as the only defense

Rejected because: The talk warns that public agents face bad actors and need multi-layer defenses.

Always send repeated queries through the agent

Rejected because: Caching can deliver the same high-quality outcome faster and cheaper for repeated queries.

Evaluate only the final output of a multi-stage RAG pipeline

Rejected because: End-to-end scores hide whether losses come from query rewrite, retrieval, reranking, or summarization.

Patterns

prompt-versioning (lifecycle)

Prompts, tools, and functions are treated as code in Git so teams can diff prompt changes and roll back agent versions.

eval-driven-development (quality)

A golden dataset of expected interactions scores runtime responses and each pipeline stage against metrics such as semantic similarity, tool calling, coherence, and fluency.

guardrails-io-filtering (safety)

The talk recommends language/category/session input filters, API security, output error handling, retries, controlled generation, and JSON-output validation.

semantic-caching (serving)

Repeated user queries can reuse cached high-quality responses instead of invoking the full agentic system.

llm-as-judge (quality)

Optimization loops use evaluation metrics from agent responses to ask another LLM system to improve prompts for coherence or tool-calling losses.

Tradeoffs

Optimize prompts with another model

Chose Meta-prompting and prompt optimization over Only hand-written prompt engineering.

Rationale: The speaker argues LLMs can describe tasks in higher fidelity than many humans and can optimize prompts from evaluation feedback.

Use cache for repeated high-quality answers

Chose Cached responses over Invoking the latest agent/model every time.

Rationale: The talk prioritizes outcome quality, latency, and token cost over always using the most advanced model path.

Evaluate pipeline stages separately

Chose Stage-level evals for query rewrite, retrieval, reranking, and summarization over Only final-response scoring.

Rationale: Stage-level metrics localize regressions and allow targeted rollback or model replacement.

Manage agent versions as code

Chose Git-based artifacts and CI/CD-style lifecycle over Manual prompt/version management in a UI.

Rationale: The speaker says prompts, functions, and tools should be diffed and rolled back like software development artifacts.

Gotchas (what bit them)

Production quality is not determined by the model alone.

Root cause: Teams over-focus on which LLM they use and under-invest in the system around it.

Resolution: Treat production agents as system-design work involving grounding, orchestration, APIs, CI/CD, analytics, safety, and evals.

Prompt injection can unfold over many turns.

Root cause: Bad actors exploit long-tail conversations rather than only a single malicious prompt.

Resolution: Apply input filters and session limits such as 30 or 50 turns as one layer of defense.

Manual response inspection does not scale after a regression.

Root cause: Without a golden dataset and metrics, teams cannot quickly compare a baseline agent with a changed agent.

Resolution: Define expectations and score runtime responses against them on semantic similarity, tool calling, coherence, and fluency.

End-to-end RAG evals can hide the failing stage.

Root cause: A new model or component may affect query rewrite, retrieval, reranking, or summarization, but the outer score only says the final response worsened.

Resolution: Evaluate each stage separately so the team can swap or fix only the stage with the largest loss.

Evaluation

The recommended evaluation model starts with a golden dataset of expected interactions, scores runtime agent behavior against metrics, feeds those metrics back into prompt optimization, and evaluates multi-stage RAG pipelines at each stage rather than only at the final output.

Metrics: semantic similarity · tool calling accuracy · coherence · fluency · query rewrite quality · retrieval quality · summarization quality

Outcomes

• The talk distills three production lessons from Google engagement work: meta-prompting, safety/guardrails, and evaluations.
• It provides operational guidance for versioning agent artifacts, using caching, and evaluating stage-level failures.
• No customer-specific quantitative outcome metrics are reported.

The speaker says the lessons come from hundreds of models delivered with developers, customers, and partners, but the talk is a lessons presentation rather than a measured case study.