Building and Scaling an Enterprise AI Assistant with GPT Models

instacart · E-commerce · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise productivity assistants

Instacart wanted to turn an engineering hackathon productivity win with GPT-4 into a secure, company-wide internal AI assistant that technical and non-technical employees could actually adopt.

Constraints:

• employees needed enterprise data privacy, security, and quota guarantees before using GPT models with internal work
• the initial engineering-oriented interface had to become usable by non-engineering teams
• blank-chat-box prompting created adoption friction for users without prompt-engineering confidence
• a separate web page was too much friction for small everyday tasks
• shared AI-generated summaries needed a lightweight way for participants to verify and correct them

Architecture

Instacart built Ava as an internal assistant powered by OpenAI GPT-4 and GPT-3.5 APIs. The web interface resembles ChatGPT but adds enterprise features: conversation search, conversation sharing, single-click code copying, keyboard shortcuts, automatic GPT-4 model upgrades as context grows, and a Prompt Exchange where employees publish reusable prompts. Slack integration brings Ava into existing workflows with link unfurling, normal chat behavior, and a first-class '@Ava summarize' feature that turns threads or channels into facts, open questions, and action items. The roadmap points toward knowledge retrieval, code execution, and company-wide Ava APIs for deeper internal tooling.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ava-web-interface[Ava web interface] --> ava-assistant-backend[Ava assistant backend]
  ava-assistant-backend --> openai-api[OpenAI API]
  ava-web-interface --> conversation-store[Conversation store]
  ava-web-interface --> prompt-exchange[Prompt Exchange]
  prompt-templates[Prompt templates] --> ava-web-interface
  prompt-exchange --> prompt-templates
  ava-assistant-backend --> slack-app[Slack app]
  slack-app --> slack-threads[Slack threads]
  slack-app --> slack-summary-verification[Slack summary verification]
  conversation-store --> slack-unfurling[Slack unfurling]
  slack-unfurling --> ava-web-interface
  ava-assistant-backend --> ava-apis[Ava APIs]
  ava-apis --> knowledge-retrieval[Knowledge retrieval]
  ava-apis --> code-execution[Code execution]

Components

Data flow

An employee uses Ava through the web UI or Slack. Web conversations can start from templates or Prompt Exchange prompts, use OpenAI GPT-4/GPT-3.5 through enterprise API guarantees, and persist for full-text search and sharing. Shared web links unfurl in Slack so colleagues can preview context. In Slack, a user can ask Ava normally or invoke '@Ava summarize' on a thread or channel; Ava generates a public summary so participants can verify accuracy and add corrections. Longer conversations can be automatically upgraded across GPT-4 model variants as context grows.

Why this architecture

Instacart reused OpenAI's enterprise APIs rather than self-hosting because early GPT-4 access came with the privacy, security, quota, and long-context guarantees needed for internal use. The product grew through adoption friction removal: engineer-specific affordances for the first launch, templates and conversation sharing for non-engineers, a crowdsourced Prompt Exchange instead of centralized template authoring, and Slack integration for tasks too small to justify visiting a separate web app. Public Slack summaries create a lightweight human-verification loop around generated thread digests.

Models used: GPT-4 · GPT-4 32K · GPT-3.5

Scale

• More than half of Instacart employees used Ava every month.
  “Ava has seen accelerated adoption at Instacart, with over half of Instacart employees using it every month and more than 900 using it every week.”article, stated
• More than 900 employees used Ava every week.
  “Ava has seen accelerated adoption at Instacart, with over half of Instacart employees using it every month and more than 900 using it every week.”article, stated
• The hackathon team produced nearly twice as many features as initially planned with ChatGPT help.
  “our team was able to produce nearly twice as many features as initially planned with the help of ChatGPT.”article, stated
• Post-launch users spent more than 20 minutes per session.
  “users were spending 20+ minutes per session and were producing and copying a significant amount of code with Ava as a chat companion.”article, stated
• By the Slack-integration stage, nearly one third of the organization already used Ava monthly.
  “By this point, nearly a third of the organization was already using Ava every month, but we wanted to reach more users and make Ava even more accessible.”article, stated

Alternatives considered (roads not taken)

Keep Ava as an engineering-only tool

Rejected because: Demand from Operations, Recruiting, Brand Marketing, HR, and other departments required a more ergonomic cross-functional assistant.

Blank text box as the default non-engineering entry point

Rejected because: Non-engineering users needed structured starting points, so Instacart added templates.

Central team authors templates for every department

Rejected because: The small Ava team lacked domain expertise for every team's use cases, so users created and shared their own prompts.

Make the Slack app a reduced web clone

Rejected because: The team instead selected Slack-native features, especially thread and channel summarization.

Standalone web app for every quick task

Rejected because: Navigating to a page and pasting context was too much friction for small tasks, so Ava moved into Slack.

Patterns

model-routing (model-management)

Ava automatically upgrades between GPT-4 models as conversation context grows, using larger context windows when needed.

prompt-versioning (prompt-operations)

The Prompt Exchange turns reusable prompts into shared organizational assets that employees can browse, search, star, create, and reuse.

human-in-the-loop-gate (collaboration)

Slack summaries are posted publicly so people in the conversation can verify the AI output and note corrections.

tool-use-orchestration (platform)

The team planned to expose Ava APIs and add knowledge retrieval plus code execution so internal teams could build AI-enabled tools on top of centralized assistant infrastructure.

Tradeoffs

Model hosting

Chose OpenAI enterprise APIs over self-hosting an internal model first.

Rationale: Early GPT-4 access plus privacy, security, quota, and 32K context support let Instacart launch quickly for engineers.

Adoption path

Chose engineering-first launch followed by cross-functional ergonomics over waiting for a fully general enterprise assistant before launch.

Rationale: Engineering use cases proved value quickly, then adoption signals from other teams drove templates, sharing, search, and Slack integration.

Prompt supply

Chose crowdsourced Prompt Exchange over small-team template creation for every department.

Rationale: Domain experts across the company could create better prompts for their own workflows than the central Ava team could.

Slack integration design

Chose Slack-native summarization and conversation support over a reduced clone of the web UI.

Rationale: Thread/channel summarization was already a proven use case and fit the Slack context better than copying all web features.

Gotchas (what bit them)

A blank chat box slowed adoption outside engineering.

Root cause: Non-technical users may not know how to craft an effective prompt from scratch.

Resolution: Add well-crafted templates to start conversations quickly.

Centralized template authoring would not scale across departments.

Root cause: The small Ava team lacked domain-specific expertise for every team's use cases.

Resolution: Launch Prompt Exchange so employees create and share prompts based on their own needs and experience.

Web-only access created too much friction for small tasks.

Root cause: Users had to navigate to a page and paste context before asking for help.

Resolution: Build a Slack app and prioritize context-native features such as thread summarization.

LLMs were limited by what data they could access.

Root cause: The model's training/tuning data and available context constrained usefulness for internal workflows.

Resolution: Prioritize knowledge retrieval and code execution so Ava can access Instacart internal knowledge and act on it.

Evaluation

The case reports adoption and engagement metrics rather than a formal evaluation harness. Instacart measured monthly and weekly usage, session duration, code-copying behavior, template usage, conversation sharing, Slack adoption, and public correction opportunities for summaries.

Metrics: monthly active employee share · weekly active employees · 20+ minutes per session · large proportion of conversations started with templates · conversation sharing and Slack visibility · summary corrections in public Slack

Outcomes

• Ava grew from a hackathon project into a company-wide internal assistant
• more than half of Instacart employees used Ava monthly and more than 900 used it weekly
• nearly a third of the organization used Ava monthly before the Slack integration push
• Prompt Exchange distributed prompt creation across the organization
• Slack summarization made Ava available inside existing work conversations
• the team planned knowledge retrieval, code execution, and company-wide APIs as the next foundation

The case reports strong adoption but not quantified productivity impact beyond usage, session, and hackathon velocity signals.