Building Product Copilots: Engineering Challenges and Best Practices

various · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production engineering for AI product copilots

The study examines how software engineers integrate LLM-powered copilots into products when normal software practices do not fit non-deterministic prompts, model orchestration, testing, safety, telemetry, compliance, and rapidly changing tooling.

Constraints:

• Most participating engineers were general software engineers encountering AI integration for the first time, not ML specialists.
• Prompt behavior is fragile, difficult to test, and often developed through trial and error.
• Production copilots require intent detection, skill routing, response interpretation, and output transformation beyond single LLM calls.
• Testing and evaluation are expensive and flaky because outputs are non-deterministic.
• Responsible AI, privacy, telemetry, and compliance requirements restrict what can be logged and shipped.

Architecture

The paper is an empirical study rather than a single product implementation. It defines product copilots as systems that translate user actions into prompts for an LLM and transform model outputs into user-facing interactions. Across interviewed teams, the common architecture includes prompt assets and templates, orchestration for intent detection and skill routing, model calls, output wrangling into formats such as JSON or product actions, safety/privacy/compliance filters, telemetry, benchmarks, and release gates. The paper's proposed tooling opportunities include prompt linters, tracing and optimization, LLM-assisted prompt writing, automated benchmark creation, cost alerting, and unified development environments.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  user-interaction-surface["User interaction surface"] --> prompt-assembly-layer["Prompt assembly layer"]
  prompt-asset-store["Prompt asset store"] --> prompt-assembly-layer
  prompt-assembly-layer --> intent-router["Intent router"]
  intent-router --> skill-orchestrator["Skill orchestrator"]
  skill-orchestrator --> llm-endpoint["LLM endpoint"]
  llm-endpoint --> output-wrangler["Output wrangler"]
  output-wrangler --> safety-and-compliance-layer["Safety and compliance layer"]
  output-wrangler --> product-action-layer["Product action layer"]
  product-action-layer --> telemetry-layer["Telemetry layer"]
  safety-and-compliance-layer --> telemetry-layer
  telemetry-layer --> benchmark-and-evaluation-harness["Benchmark and evaluation harness"]
  benchmark-and-evaluation-harness --> release-gate["Release gate"]
  benchmark-and-evaluation-harness --> prompt-quality-tooling["Prompt quality tooling"]
  prompt-quality-tooling --> prompt-asset-store

Components

Data flow

User actions or natural-language requests enter a product copilot. The system maps the request into prompts, assembles prompt assets and context, routes the intent to a skill or workflow, calls an LLM, parses or transforms the output into a product action or response, and then subjects the result to validation, safety/privacy controls, telemetry, and evaluation. Feedback and benchmark data loop back into prompt changes, test suites, and tool improvements.

Why this architecture

The study shows that production copilots need explicit engineering layers around the LLM because prompts, raw outputs, and ad hoc playground workflows do not provide enough reliability. Intent routing, output wrangling, safety review, telemetry, and benchmark loops are needed because copilots act inside products and because non-deterministic model behavior strains standard software engineering practices.

Models used: Large Language Models · GPT-4 · Internally hosted OpenAI models · Managed endpoint models

Scale

• The study interviewed 26 professional software engineers.
  “interview study with 26 professional software engineers responsible for building product copilots”article, stated
• Fourteen participants were internal Microsoft engineers.
  “internal Microsoft engineers working on publicly announced Copilot products (14 participants)”summary, stated
• Twelve participants were external engineers from various companies.
  “external engineers from various companies recruited via UserInterviews.com (12 participants)”summary, stated
• Interviews lasted 45 minutes each.
  “semi-structured interviews (45 minutes each)”summary, stated
• Some teams ran each test ten times and required seven successes.
  “running "each test 10 times" and only considering it passed if 7 of 10 instances succeeded”summary, stated
• One team externally labeled about 10,000 responses.
  “One team labeled approximately 10,000 responses externally”summary, stated
• LLM benchmark calls could cost 1-2 cents per test.
  “each test would probably cost 1-2 cents to run”summary, stated
• Responsible AI assessment could require multiple weeks of work.
  “requiring multiple weeks of documentation and assessment work”summary, stated
• One team needed automated benchmarks over hundreds of harm subcategories.
  “automated benchmarks covering hundreds of subcategories of potential harm before shipping”summary, stated

Alternatives considered (roads not taken)

Ad hoc playground-based prompt engineering.

Rejected because: Engineers described this as time-consuming trial and error without structured guidance or integrated tooling.

Forcing rigid output formats regardless of model tendencies.

Rejected because: The study found that models often malformed requested formats and that working with natural output tendencies could perform better.

Exact-output unit tests for non-deterministic copilots.

Rejected because: Model outputs vary, so teams used repeated runs, acceptable-output sets, metamorphic tests, or graded evaluations instead.

Open-ended product action access from natural language.

Rejected because: Users expected broad commanding, but safety concerns and engineering effort limited what copilots could safely execute.

Using LangChain-style frameworks directly for production.

Rejected because: Participants found such frameworks useful for prototypes but often avoided them in products due to learning curve and customer-problem focus.

Patterns

prompt-versioning (prompt_ops)

Teams kept growing prompt libraries in version control, but lacked systematic performance tracking for changes.

tool-use-orchestration (agent_architecture)

Product copilots use intent detection and skill routing to connect user requests to product capabilities.

eval-driven-development (evaluation)

Teams used repeated tests, spreadsheets, manually labeled examples, benchmarks, grades, and structural tests to cope with non-deterministic output.

safety-policy-enforcement (safety)

Teams layered rule-based classifiers, blocklists, privacy checks, and responsible AI assessments around managed model endpoints.

observability-tracing (operations)

The study identifies visibility into agent behavior and telemetry as a tooling gap, complicated by privacy limits on logging prompts.

cost-aware-routing (operations)

Benchmarking and prompt changes need cost awareness because repeated LLM endpoint tests can add up and affect production endpoints.

structured-output-contract (output_control)

Teams tried JSON schemas and output wrangling to get machine-readable results, but the study shows this remains brittle without tooling.

context-compaction (context_management)

Teams struggled to select and truncate context while understanding how each prompt component affected performance.

Tradeoffs

Treat prompt engineering as a managed software artifact.

Chose Prompt assets, version control, linters, tracing, and benchmark feedback over Ad hoc playground experimentation.

Rationale: Prompt libraries grew complex enough that teams needed ways to validate and track changes over time.

Route copilots through skills and orchestration.

Chose Intent detection, skill routing, and post-response processing over Single-turn LLM calls directly attached to product UI.

Rationale: Production product copilots have to interpret intent, choose capabilities, and transform outputs into product-specific behavior.

Use property, repeated-run, and graded evaluation methods.

Chose Probabilistic and structural testing over Deterministic exact-match unit tests only.

Rationale: Non-deterministic model outputs require evaluation methods that account for acceptable variation.

Add safety and compliance layers beyond managed endpoint filters.

Chose Rule-based classifiers, blocklists, privacy processing, and responsible AI assessments over Relying only on model endpoint safety filters.

Rationale: Managed filters were insufficient for product-specific harms, privacy, and compliance requirements.

Limit telemetry to privacy-compliant signals.

Chose Skill-use and aggregate behavior visibility over Logging raw user prompts.

Rationale: Privacy constraints blocked raw prompt logging, even though teams needed more observability to understand interactions.

Gotchas (what bit them)

Prompt engineering behaved more like ad hoc experimentation than normal software engineering.

Root cause: Engineers lacked structured tools and guidance, so they moved between playgrounds and prompt experiments by trial and error.

Resolution: The paper proposes prompt linters, prompt tracing/optimization, and LLM-assisted prompt writing to make prompt work more systematic.

Machine-readable output contracts were brittle.

Root cause: Models could produce malformed output, hallucinate stop tokens, or vary formatting despite schema instructions.

Resolution: Use output wrangling and prefer formats aligned with the model's natural tendencies where possible.

Agentic workflows were hard to manage and steer.

Root cause: Models could fail to recognize completion, get stuck in loops, or go far off track during multi-turn work.

Resolution: Treat agent orchestration as a separate engineering concern with visibility, routing, and guardrails rather than simple prompt chaining.

Flaky tests made conventional test pass/fail logic unreliable.

Root cause: Each model response can differ, so an exact assertion can fail for reasons unrelated to product quality.

Resolution: Use repeated runs, multiple acceptable outputs, metamorphic properties, grading schemes, and benchmark datasets.

Benchmark creation was expensive and labor-intensive.

Root cause: No standardized benchmarks existed, so each team had to curate or label its own datasets.

Resolution: The paper proposes automated benchmark creation, including converting user feedback into benchmarks.

Privacy constraints limited telemetry exactly where teams needed visibility.

Root cause: Teams needed to understand user interactions but could not log prompts because of privacy and compliance restrictions.

Resolution: Log safer interaction metadata such as skill use, while developing privacy-aware observability patterns.

Evaluation

The paper itself uses mixed methods: semi-structured interviews plus structured brainstorming with 26 engineers. The copilots studied used repeated-run tests, curated input/output spreadsheets, multiple acceptable outputs, metamorphic tests, manually labeled benchmarks, external labeling, simple grading schemes, and responsible AI benchmark suites.

Metrics: 26 interviewed professional engineers · 14 internal Microsoft participants and 12 external participants · 45-minute semi-structured interviews · Repeated test success such as 7 of 10 runs · Hundreds of curated input/output examples · Approximately 10,000 externally labeled responses in one team · Per-test LLM cost of roughly 1-2 cents · Hundreds of harm subcategories for one responsible AI benchmark

Outcomes

• The study documents pain points across prompt engineering, orchestration, testing, safety, privacy, telemetry, and tooling.
• It identifies brittle prompts, output wrangling, context management, prompt asset management, flaky tests, benchmark costs, and responsible AI review as recurring production issues.
• It proposes prompt linters, prompt tracing, LLM-assisted prompt writing, automated benchmark creation, cost alerting, and unified development workflows as tooling opportunities.
• It provides empirical grounding for why product-copilot engineering needs new tools rather than direct reuse of traditional software practices.

The study covers 26 engineers, 14 internal and 12 external, through 45-minute interviews and brainstorming. Operational scale signals include hundreds of curated examples, 10 repeated test runs with a 7-of-10 passing rule, about 10,000 externally labeled responses for one team, 1-2 cents per benchmark test, multiple weeks of responsible AI documentation, and hundreds of harm subcategories in one benchmark.