Best Practices for AI Agent Development and Deployment

microsoft · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production AI agent design

Microsoft's agent-development guidance addresses teams that want to deploy production agents but risk giving them too much agency, too broad an environment, or tool choices before defining goals and constraints.

Constraints:

• agents act autonomously and can take actions on users' behalf
• raw web access and broad action spaces are unsafe starting points
• MVPs should keep humans in the loop
• the first step is defining the environment, end goal, intended behavior, and capabilities
• production agent workflows face speed/reliability tradeoffs

Architecture

The guidance recommends designing agents from first principles: define the agent's environment, goal, intended behavior, capabilities, and toolset on paper before coding. Start in a constrained state space, use off-the-shelf/no-code/function-calling tools before custom orchestration, keep humans in the loop for MVP actions, make the agent explain or simulate next steps before acting, and mix LLM steps with smaller models, traditional ML, regex, or synchronous/asynchronous workflow steps when those are more reliable or efficient.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  agent-goal[Agent goal] --> environment-definition[Environment definition]
  agent-goal --> tool-set[Tool set]
  environment-definition --> constrained-mvp[Constrained MVP]
  tool-set --> constrained-mvp
  constrained-mvp --> human-approval-gate[Human approval gate]
  human-approval-gate --> production-workflow[Production workflow]
  production-workflow --> monitoring[Monitoring]

Components

Data flow

A team first writes the agent's goal, environment, intended behavior, and available capabilities. It then gives the agent a small constrained task space, curated tools, and human approval gates. The workflow can include LLM calls, smaller language models, traditional ML, regex, synchronous or asynchronous orchestration steps, and production monitoring as reliability and latency are evaluated.

Why this architecture

The guidance treats an agent as an autonomous actor, so production design starts with action boundaries and purpose rather than framework choice. Human gates and constrained environments reduce the damage radius while teams learn whether the agent is reliable, and mixed orchestration avoids forcing every step through an LLM.

Models used: Azure OpenAI · small language models · traditional machine learning

Scale

• The guidance draws on prior experience building a deep reinforcement learning platform as a service at Microsoft.
  “product team that led developing the first deep you know um deep reinforcement learning platform as a service for my roft like the first one”transcript, stated

Alternatives considered (roads not taken)

Give the agent raw web access immediately

Rejected because: The speaker explicitly calls that a bad starting point regardless of safeguards.

Skip directly from pen-and-paper MVP to custom orchestration code

Rejected because: The recommendation is to use off-the-shelf/function/no-code tools first and learn why they fail.

Use an LLM call for every workflow step

Rejected because: Some steps can be regular regex, traditional ML, or other optimized components.

Patterns

human-in-the-loop-gate (governance)

The guidance recommends keeping humans in the loop for MVP agent actions and asking agents to explain next steps before acting.

tool-use-orchestration (agent-design)

Agents are defined by the tools/capabilities they can use to accomplish goals.

guardrails-io-filtering (safety)

The agent starts in a constrained environment rather than a broad state/action space.

latency-optimization (production)

The guidance calls out the speed/reliability tradeoff in production agent workflows.

Tradeoffs

Start with human-gated MVPs

Chose agent explains/simulates before action over unreviewed autonomous action.

Rationale: Autonomous action is the risk, not the agent producing text.

Use mixed orchestration

Chose LLMs plus SLMs, traditional ML, regex, synchronous/asynchronous steps over LLM-only workflows.

Rationale: Each step should use the most reliable and optimized mechanism.

Constrain before broadening

Chose small landing space over broad state/action space from day one.

Rationale: Testing in a bounded environment reveals behavior before high-risk expansion.

Gotchas (what bit them)

Too much agency at MVP time is dangerous.

Root cause: Agents act autonomously and can take real actions on behalf of users.

Resolution: Start with constrained environments and human approval gates.

Broad support-agent context can create failure.

Root cause: Giving an agent the entire support corpus and live tickets before testing behavior creates too much state space.

Resolution: Cherry-pick lower-priority, clearly defined cases with bounded documentation first.

Framework/tool novelty can distract from solving the problem.

Root cause: Teams jump to custom chains before understanding how to action success fastest.

Resolution: Try simpler available tools and learn why they do or do not solve the problem.

Evaluation

The guidance evaluates agents through constrained MVP behavior, human review of proposed next actions, simulation/debug streams, support-case closure quality, production monitoring, and explicit speed/reliability tradeoff observation.

Metrics: human-reviewed next-step explanations · NPS or ticket-closure quality for bounded support cases · speed vs reliability/accuracy tradeoff · success in small constrained environments before expansion

Outcomes

• The talk provides a production-agent design checklist rather than a single deployed system.
• Its central outcome is a disciplined MVP path: define environment and goal, constrain state space, keep humans in loop, then broaden.
• It argues agents should use tools and mixed orchestration, not unbounded LLM calls or raw web access.
• Production readiness depends on observed behavior, monitoring, and speed/reliability tradeoffs.

Guidance case: one concrete scale reference is prior Microsoft DRL platform work; most evidence is process-oriented rather than deployment-volume metrics.