Building an AI Co-pilot for Product Strategy with LLM Integration Patterns

thoughtworks · Consulting · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.0 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Consulting

Thoughtworks explored how an AI co-pilot could support product strategy activities such as scenario building, ideation, concept generation, and storyboarding.

Constraints:

• users should be able to start from naive prompts
• LLM output needed to be presented as structured UI elements
• follow-up interactions needed to preserve selected context
• external research needed to be incorporated into ideation

Architecture

Boba is a web application that mediates product-strategy tasks between a human user and OpenAI GPT-3.5/4. The task UI captures user prompts and selections, LangChain prompt templates enrich them with task-specific context, OpenAI returns structured JSON/Markdown outputs, and the UI renders scenario cards, concepts, strategies, or storyboards. Follow-up interactions carry selected UI context into contextual conversations. Research Signals adds external knowledge by querying Google SERP, fetching articles, chunking them into an in-memory HNSW vector store, and using LangChain VectorDBQAChain to answer with source-linked context. Streaming tokens back to the UI gives progress and cancellation for long generations.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  boba-web-application[Boba web application] --> task-ui[Task UI]
  task-ui --> langchain-prompt-templates[LangChain prompt templates]
  langchain-prompt-templates --> openai-llm[OpenAI LLM]
  openai-llm --> structured-response-renderer[Structured response renderer]
  openai-llm --> streaming-token-handler[Streaming token handler]
  structured-response-renderer --> task-ui
  task-ui --> select-and-carry-context[Select and carry context]
  select-and-carry-context --> contextual-conversation[Contextual conversation]
  contextual-conversation --> langchain-prompt-templates
  google-serp-api[Google SERP API] --> embedded-external-knowledge[Embedded external knowledge]
  embedded-external-knowledge --> hnsw-vector-store[HNSW vector store]
  hnsw-vector-store --> vectordbqa-chain[VectorDBQAChain]
  vectordbqa-chain --> openai-llm
  task-ui --> stable-diffusion-storyboarding[Stable Diffusion storyboarding]
  stable-diffusion-storyboarding --> openai-llm

Components

Data flow

A user selects a product-strategy task in the Boba web UI and enters a naive prompt plus task controls such as time horizon or realism. Boba fills a task-specific LangChain prompt template, sends it to OpenAI GPT-3.5/4, requests structured output, and renders the result as UI elements. If the user selects a scenario or concept, Boba carries that selection into a new contextual prompt. For research questions, Boba sends an enriched query to Google SERP, retrieves articles, summarizes or chunks them, creates embeddings/vector-store entries, and prefixes retrieved article context into the LLM prompt so answers include source links.

Why this architecture

Boba’s architecture exists to move beyond a plain chat box. The UI structures tasks so users can make naive prompts while the application supplies prompt context, output schemas, selected-context carry-forward, progress feedback, and retrieval/tooling when the base LLM is insufficient. The team kept prompt templates simple, used structured output so the UI could operate on LLM results, streamed long responses to keep users engaged, and controlled external search flow manually where LangChain tool autonomy produced slow or harder-to-parse results.

Models used: GPT-3.5 · GPT-4 · Stable Diffusion

Scale

• Boba is explicitly an experimental prototype rather than a production scale report.
  “Boba is an experimental AI co-pilot for product strategy & generative ideation”article, stated
• The article says most effort went into UI/prompt engineering rather than model work.
  “we also found that we spent way more time on the user interface (about 80%) than the AI itself (about 20%), specifically in engineering the prompts.”article, stated
• The research feature uses an in-memory vector store for the prototype and recommends external vector DBs for larger or long-term memory use cases.
  “we used an in-memory vector store called HNSWLib (Hierarchical Navigable Small World).”article, stated

  “for larger scale use cases and/or long-term memory, we recommend using an external vector DB like Pinecone or Weaviate.”article, stated

Alternatives considered (roads not taken)

A simple web chat front-end to an LLM

Rejected because: Users would need to learn prompt engineering themselves; Boba structures task controls and enriches prompts for them.

Complex conditional prompt templates

Rejected because: The team preferred separate templates when inputs changed drastically, keeping templates maintainable.

Waiting for full LLM responses behind a spinner

Rejected because: Users lose patience on long calls, so Boba streams partial output and allows cancellation.

Using LangChain external tools API to run Google search autonomously

Rejected because: It delegated too much decision-making and produced mixed, slow, harder-to-parse results.

Patterns

prompt-chaining (workflow)

Boba enriches a user prompt for scenario generation, then lets the user select a scenario and continue with a context-aware follow-up prompt.

structured-output-contract (interface)

The application expects structured LLM responses that can be rendered as UI elements, not only free-form chat text.

rag-retrieval (retrieval)

Boba uses search results as external knowledge before asking the LLM to summarize articles for product-strategy research.

Tradeoffs

How to make product-strategy LLM interactions usable for non-prompt-experts.

Chose task-specific UI controls plus templated prompt enrichment over a plain chat UI that leaves users to craft prompts manually.

Rationale: The UI can capture structured intent and context, letting users make naive prompts while Boba enriches them for better LLM responses.

How to keep prompt templates maintainable.

Chose separate prompt templates for drastically different inputs over complex conditional logic inside one prompt template.

Rationale: The team found simple templates easier to maintain and swapped templates when UI context changed meaningfully.

How to implement external research search flow.

Chose explicit SERP + article processing + VectorDBQAChain flow over letting LangChain external tools API decide the whole search process.

Rationale: Manual orchestration gave more predictable and parseable behavior for search-backed research answers.

Evaluation

No grounded evaluation method is stated in the available input.

Outcomes

The source reads as an exploratory implementation-pattern article rather than a production scale report.