Building an AI Co-Pilot Application: Patterns and Best Practices

thoughtworks · Consulting · 2023 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI co-pilot product design

Thoughtworks wanted to understand how to build LLM-powered co-pilot applications that mediate between users and models instead of exposing a plain chat interface.

Constraints:

• Users may not be skilled prompt engineers.
• The app needed structured model outputs for downstream UI behavior.
• The team needed to stream progress and manage user context across interactions.
• The application needed external knowledge beyond the base model cutoff.
• Most effort still went into UX and application engineering.

Architecture

Boba is a web co-pilot that sits between users and GPT-3.5/4. The frontend captures product-strategy inputs; the backend uses LangChain prompt templates, structured JSON responses, streaming, user-selected context, vector-store retrieval, web search, and image generation to support product ideation workflows.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  boba_web_frontend["Boba web frontend"] --> backend_orchestration_service["Backend orchestration service"]
  backend_orchestration_service --> prompt_templates["Prompt templates"]
  prompt_templates --> openai_gpt_models["OpenAI GPT models"]
  backend_orchestration_service --> vector_store["Vector store"]
  external_knowledge_apis["External knowledge APIs"] --> vector_store
  vector_store --> backend_orchestration_service
  backend_orchestration_service --> streaming_progress_channel["Streaming progress channel"]
  streaming_progress_channel --> boba_web_frontend
  backend_orchestration_service --> image_generation_service["Image generation service"]

Components

Data flow

A user enters product-strategy intent and UI selections. Boba enriches that input through templates and persona instructions, calls GPT models, parses structured JSON responses, streams progress to the UI, carries selected context forward, and optionally retrieves external knowledge through embeddings, vector search, and web/article APIs.

Why this architecture

The architecture turns LLM access into a product-specific co-pilot by adding UI affordances, templates, structured outputs, retrieval, and context-carrying patterns around the base model.

Models used: GPT-3.5 · GPT-4 · OpenAI Embeddings · Stable Diffusion

Scale

• The team reported that about 80% of development effort went into UI and 20% into AI/prompt engineering.
  “approximately 80% of effort went into user interface development, while only 20% went into the AI/prompt engineering aspects.”summary, stated
• The case documents eight reusable co-pilot application patterns.
  “The project identified eight key patterns for mediating conversations between users and LLMs”summary, stated

Alternatives considered (roads not taken)

Simple user prompt passthrough

Rejected because: The team enriched short user inputs with templates, persona, and UI-controlled context.

Free-form model output only

Rejected because: Production applications need structured data that can be parsed and rendered.

Rely only on model pretraining

Rejected because: The team added embedded external knowledge with RAG to overcome knowledge cutoff limits.

Patterns

prompt-versioning (prompt design)

Boba uses prompt templates as reusable artifacts for different product strategy workflows.

structured-output-contract (output parsing)

The system asks GPT to return well-formed JSON for downstream UI and processing.

rag-retrieval (external knowledge)

The co-pilot integrates external knowledge through embeddings, vector stores, web search, and article extraction.

human-in-the-loop-gate (user steering)

The UI lets users select and carry context forward rather than letting the model autonomously decide all context.

Tradeoffs

Build application-specific mediation around GPT.

Chose A co-pilot with templates, structured responses, context carrying, and retrieval over A generic chat wrapper around the model.

Rationale: Users need workflow support and domain context, not just model access.

Keep prompt templates simple.

Chose Different template files for substantially different use cases over Complex conditional logic inside templates.

Rationale: Simple templates are easier to reason about and maintain.

Gotchas (what bit them)

Production co-pilots are mostly product engineering, not only prompt engineering.

Root cause: The usable application needs UI, context controls, streaming, parsing, and workflow design around the model.

Resolution: Treat the model as one component in a product-specific co-pilot architecture.

Structured response quality depends on prompting examples and schema clarity.

Root cause: The model needs enough output-shape context to produce parseable JSON.

Resolution: Describe schemas in prompts and provide few-shot output examples.

Evaluation

The case evaluates Boba through documented design patterns and practical implementation observations rather than a benchmark; the strongest quantitative observation is engineering effort allocation.

Metrics: Eight documented co-pilot patterns. · Approximately 80% UI effort and 20% AI/prompt engineering effort.

Outcomes

• Thoughtworks built Boba as an experimental product-strategy co-pilot.
• The team documented eight reusable LLM application patterns.
• The case showed that user experience and workflow mediation dominate production co-pilot work.

This is an experimental co-pilot case; the main quantified scale fact is development effort allocation.