Challenges and Opportunities in Building Product Copilots: An Industry Interview Study

microsoft_/_github · Tech · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Engineering lifecycle for AI product copilots

Microsoft and GitHub researchers found that teams building product copilots were improvising across prompt engineering, orchestration, testing, safety, and developer tooling because standard software-engineering processes had not caught up with LLM-powered product development.

Constraints:

• The case is an interview study rather than a single production deployment.
• Product copilots need to respond to natural-language requests in user-specific product context.
• Engineers face nondeterministic model behavior, fragile prompts, multi-turn orchestration, and weak benchmark coverage.
• Safety, privacy, and compliance requirements must be handled without mature standard tooling.

Architecture

The paper maps a product-copilot engineering lifecycle: user-contextual prompts feed orchestration and multi-turn workflows; teams manage prompt assets, logging, test runs, and benchmarks; safety/privacy/compliance constraints wrap deployment; and proposed tools such as prompt linters, automated benchmark generation, tracing, telemetry, and integrated development workflows close the loop.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  product-copilot-interface["product copilot interface"] --> prompt-template-layer["prompt template layer"]
  product-copilot-interface --> orchestration-workflow["orchestration workflow"]
  prompt-template-layer --> orchestration-workflow
  orchestration-workflow --> llm-service["LLM service"]
  orchestration-workflow --> tool-or-product-apis["tool or product APIs"]
  orchestration-workflow --> guardrail-layer["guardrail layer"]
  llm-service --> guardrail-layer
  prompt-template-layer --> prompt-observability-logs["prompt observability logs"]
  prompt-observability-logs --> benchmark-and-evaluation-suite["benchmark and evaluation suite"]
  guardrail-layer --> benchmark-and-evaluation-suite
  benchmark-and-evaluation-suite --> developer-tooling["developer tooling"]

Components

Data flow

A product copilot receives a user request in product context, passes through prompt templates and orchestration logic, may perform intent detection and multi-step tool/workflow calls, returns a response, and then logs prompt/output behavior into testing, benchmarking, telemetry, safety review, and prompt-improvement workflows.

Why this architecture

The study argues that copilots cannot be engineered as isolated prompts: teams need lifecycle tooling because prompts are dynamically assembled, workflows are multi-step, model behavior is nondeterministic, and traditional tests are too brittle on their own.

Models used: LLMs for product copilots · GPT-4 as a rubberduck for prompt work

Scale

• The study is grounded in interviews with 26 professional software engineers.
  “an interview study with 26 professional software engineers responsible for building product copilots at various companies.”article, stated
• The arXiv paper is 11 pages and was submitted in December 2023.

  "Comments:

11 pages"

• One testing workaround described running each test 10 times and accepting 7 of 10 passes.
  “running "each test 10 times" and considering it passing only if "7 of the 10 instances passed."”summary, stated

Alternatives considered (roads not taken)

Treat copilots as ordinary software features with existing engineering tools

Rejected because: The study found pain points across the whole engineering process because existing tools had not adapted to LLM-specific behavior.

Manage prompts only with version control

Rejected because: Versioned prompt assets did not provide continuous validation, performance tracking, or visibility into dynamic prompt assembly.

Use traditional deterministic unit tests for LLM behavior

Rejected because: Engineers found LLM tests flaky because model responses can vary on repeated runs.

Build copilot workflows as single prompts

Rejected because: Production copilots required orchestration across prompts, intents, components, and multi-step tasks.

Patterns

prompt-versioning (prompt-engineering)

Teams kept prompt assets in version control but still lacked continuous validation and performance tracking for prompt changes.

prompt-chaining (orchestration)

Production copilots used multiple prompts and intent-detection stages rather than a single prompt-response call.

eval-driven-development (evaluation)

The study identifies benchmarks, repeated test runs, telemetry, and automated benchmark creation as necessary supports for copilot development.

safety-policy-enforcement (safety)

Safety, privacy, compliance, and guardrails appear as core constraints in product-copilot engineering rather than post-deployment concerns.

observability-tracing (operations)

Teams needed visibility into final prompt construction, dynamic steps, model behavior, and multi-step tasks to debug production copilots.

Tradeoffs

Use repeated stochastic test runs.

Chose Run each test multiple times and accept pass-rate thresholds over Expect a deterministic pass/fail assertion from one model call.

Rationale: LLM responses vary, so repeated trials better reflect behavior than a single brittle assertion.

Invest in integrated copilot development tooling.

Chose Prompt, orchestration, testing, telemetry, and benchmark tools in one workflow over Fragmented ad hoc scripts and traditional software-only tools.

Rationale: The paper finds friction across every stage, so isolated fixes do not address lifecycle-wide LLMOps needs.

Use model assistance for prompt debugging.

Chose GPT-4 as a rubberduck for prompt work over Manual prompt debugging only.

Rationale: The proposed solutions include using a stronger model to reason about prompt behavior, but only as support for engineer judgment.

Gotchas (what bit them)

Prompt changes are hard to reason about once prompts are dynamically assembled.

Root cause: The final prompt can combine templates, examples, rules, and dynamic steps that are hard to inspect after the fact.

Resolution: Add prompt tracing, linters, and change-impact tooling rather than relying on static prompt files alone.

Classical tests become flaky for LLM-powered systems.

Root cause: Nondeterministic model responses undermine exact assertions and make one-scenario tests poor predictors.

Resolution: Use repeated runs, probabilistic pass criteria, benchmarks, and automated benchmark creation.

Multi-step copilots can lose the task thread.

Root cause: Long-running sessions and internal orchestration states are opaque, so a model may misinterpret a user turn as completion.

Resolution: Improve tracking of multi-step tasks and add guardrails around agent behavior.

Teams lack shared best practices during rapid technology change.

Root cause: LLM application engineering was new enough that existing software practices and tooling did not cover the work.

Resolution: Build integrated development workflows and shared tooling for prompts, orchestration, tests, telemetry, and safety.

Evaluation

The paper itself evaluates engineering needs through interviews and brainstorming; the copilot systems studied struggled with flaky tests, weak benchmarks, repeated-run heuristics, telemetry, and prompt-change tracing.

Metrics: 26 engineer interviews · pain points across every engineering step · test pass rate over repeated LLM runs · benchmark coverage · prompt-change impact · visibility into model behavior and multi-step task state

Outcomes

• The study identifies lifecycle-wide pain points in production product-copilot engineering.
• It proposes tooling directions including prompt linters, automated benchmark creation, tracing, telemetry, and integrated workflows.
• The case is research evidence about engineering practice, not a measured deployment outcome for one copilot.

The study reports 26 professional software-engineer interviews, an 11-page arXiv paper, and group brainstorming sessions; it does not provide production traffic, user, or latency metrics for a single copilot.