Building and Sunsetting Ada: An Internal LLM-Powered Chatbot Assistant

leboncoin · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal enterprise LLM assistant

Leboncoin needed a secure internal GenAI assistant that let employees and ML teams experiment with LLMs without leaking confidential data to public services, then had to evolve that assistant into domain-specific knowledge tools and eventually decide whether maintaining it still made sense once commercial enterprise platforms met EU data-residency requirements.

Constraints:

• French and European data-residency requirements made public chatbot use risky for sensitive employee and client information
• self-hosting promised data control but created deployment, cost, scaling, and quality burdens
• privacy choices such as storing no conversations limited operational debugging and usage analysis
• different internal knowledge sources had different structures and retrieval needs
• evaluation needed to catch hallucinations, weak retrieval, multilingual failure modes, and table-structure loss
• a small ML team could not scale every team's assistant demand while maintaining custom infrastructure

Architecture

Ada began as a private Claude-powered internal chatbot on AWS Bedrock after self-hosted Llama 2 proved costly and weaker for conversation. It then expanded into multiple specialized assistants. Confluence and Lumapps used classic RAG with chunks, embeddings, Postgres vector storage, reranking, and, where useful, query rephrasing. Backstage reused its existing OpenSearch index with lexical search, a keyword rephraser, and reranker. Policy and org-chart assistants skipped retrieval by loading small corpora or the full org graph into context. Slack Apps brought assistants into employee workflows, while LangSmith, Airflow, custom datasets, correctness scoring, and correct-link metrics supported iteration. Ada was later phased out in favor of ChatGPT Enterprise with EU data residency, with some capabilities moving to MCP connectors, Custom GPTs, and n8n workflows.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ada-web-assistant[Ada web assistant] --> aws-bedrock-claude[AWS Bedrock Claude]
  ada-web-assistant --> privacy-layer[Privacy layer]
  ada-web-assistant --> assistant-router[Assistant router]
  ada-web-assistant --> slack-apps[Slack Apps]
  assistant-router --> classic-rag-assistants[Classic RAG assistants]
  assistant-router --> backstage-assistant[Backstage assistant]
  assistant-router --> full-context-assistants[Full-context assistants]
  classic-rag-assistants --> postgres-vectordb[Postgres vectorDB]
  classic-rag-assistants --> query-rephraser[Query rephraser]
  classic-rag-assistants --> cohere-reranker[Cohere reranker]
  classic-rag-assistants --> aws-bedrock-claude
  backstage-assistant --> opensearch-index[OpenSearch index]
  backstage-assistant --> query-rephraser
  backstage-assistant --> cohere-reranker
  backstage-assistant --> aws-bedrock-claude
  full-context-assistants --> organization-graph[Organization graph]
  full-context-assistants --> aws-bedrock-claude
  classic-rag-assistants --> evaluation-framework[Evaluation framework]
  backstage-assistant --> evaluation-framework
  full-context-assistants --> evaluation-framework
  chatgpt-enterprise-migration[ChatGPT Enterprise migration] --> mcp-connectors[MCP connectors]
  chatgpt-enterprise-migration --> custom-gpts[Custom GPTs]
  chatgpt-enterprise-migration --> n8n-workflows[n8n workflows]

Components

Data flow

An employee asks Ada a question through the web interface or Slack. The request is routed to a source-specific assistant. Classic documentation assistants retrieve chunks from embedded Postgres vector stores, then rerank context before Claude answers. Backstage questions are converted to English keywords, searched against OpenSearch, reranked, and answered. Policy or organization-chart questions can load the full small corpus or graph into Claude's context instead of retrieving. Evaluation datasets and LangSmith runs measure correctness and source-link retrieval, while Airflow schedules weekly baselines. When ChatGPT Enterprise added EU data residency, the internal assistant foundation shifted toward OpenAI's platform, with Ada-specific capabilities ported via MCP connectors, Custom GPT actions, and n8n automation.

Why this architecture

Ada's architecture followed the team's security and learning goals. Bedrock Claude gave better conversational quality and lower operational burden than self-hosting while preserving data controls. Specialized assistants replaced one generic chatbot because internal sources differed: some needed semantic vector retrieval, Backstage already had good lexical infrastructure, and small policy/org corpora could fit directly in context. Slack integration reduced workflow friction. The final migration was architectural pragmatism: once ChatGPT Enterprise satisfied EU data residency, the team could reuse commercial platform capability and redirect scarce GenAI expertise toward user-facing work instead of running a custom internal assistant.

Models used: Meta Llama 2 · Anthropic Claude 2 · Anthropic Claude models · Claude Sonnet · Cohere Rerank 3.5 · ChatGPT Enterprise

Scale

• A few MLEs began Ada at the end of 2023.
  “A few MLEs kicked off work at the end of 2023 on what would become Ada”article, stated
• The dedicated Ada team had three MLEs, two software engineers, a PO, and a manager.
  “a dedicated team was spun up, composed of three MLE and two software engineers (plus a PO and manager)”article, stated
• The Backstage OpenSearch rephraser improved context relevance from 0.63 to 0.73.
  “raising context relevance of 0.63 to 0.73”article, stated
• The final Backstage setup returned the correct source link in 70% of cases.
  “returning the correct source link in 70% of cases”article, stated
• LangSmith asynchronous evaluation cut a 120-example dataset run from 30 minutes to 3 minutes.
  “reducing the time for evaluating a dataset of 120 examples from 30min to 3min”article, stated
• Cohere Rerank 3.5 had a 4,096-token limit compared with Claude Sonnet's 200K-token context window.
  “Claude Sonnet, with a 200K token window), the reranker model we used — Cohere Rerank 3.5 — has a 4,096-token limit”article, stated
• A simple rule-based moderation rephraser improved correctness by 10%, while a more complex LLM rephraser improved only 3-4%.
  “boosted correctness by 10%, while a more complex rephraser (LLM-based), was only able to achieve a 3–4% improvement”article, stated
• Ada was phased out at the end of Q1 2025 after roughly a year and a half of learning.
  “at the end of the first quarter of 2025 the decision was made: Ada would be gradually phased out”article, stated

  “Ada’s journey wasn’t long — about a year and a half–but taught us”article, stated

Alternatives considered (roads not taken)

Self-host Meta Llama 2

Rejected because: Self-hosting gave theoretical data control, but deployment was complex, chat performance lagged, scaling was tricky, and cost exceeded Bedrock.

Classic RAG for every assistant

Rejected because: Backstage already had OpenSearch, and the policy/org corpora were small enough for full-context approaches.

Vector database for Backstage technical docs

Rejected because: Using existing OpenSearch avoided a new ingestion pipeline, vector database management, and embedding costs.

Use standard retrieval metrics unchanged

Rejected because: Cosine similarity, answer relevance, and context relevance did not catch hallucinations or weak retrieval in practice.

Keep maintaining Ada as the internal assistant foundation

Rejected because: ChatGPT Enterprise met EU data residency needs, and operating Ada consumed technical and operational effort better spent on user-facing GenAI work.

Patterns

privacy-preserving-processing (privacy)

Ada was built to keep employee prompts and sensitive corporate data inside controlled infrastructure, with no conversations stored by the development team.

rag-retrieval (retrieval)

Confluence and Lumapps assistants used classic RAG with chunks, embeddings, Postgres vector storage, semantic retrieval, and Claude generation.

retrieval-ranking (retrieval)

Leboncoin added Cohere Rerank 3.5 after first-pass retrieval to improve context relevance and answer reliability.

query-rewriting (retrieval)

Query rephrasers stripped noisy terms or translated multilingual queries before search, but were applied only where retrieval failure justified them.

llm-as-judge (evaluation)

Correctness used an LLM judge to compare generated answers with ground truth and became the primary quality metric.

eval-driven-development (quality)

The team built tailored datasets, custom metrics, LangSmith async evaluations, and weekly Airflow baselines to drive iteration.

tool-use-orchestration (integration)

Ada moved from a web interface into Slack Apps and then toward MCP connectors, Custom GPTs, and n8n workflows during migration.

human-in-the-loop-gate (quality)

User feedback was incorporated into evaluation datasets so real-world failures influenced future quality checks.

Tradeoffs

Model hosting

Chose Claude via AWS Bedrock over self-hosted Llama 2.

Rationale: Bedrock Claude delivered better conversational quality and reliability with a more sustainable cost model while still satisfying legal/security data guarantees.

Knowledge-source architecture

Chose source-specific assistants over one generic RAG architecture.

Rationale: Confluence, Lumapps, Backstage, policies, and org charts had different structures, language requirements, corpus sizes, and retrieval failure modes.

Backstage retrieval

Chose OpenSearch plus keyword rephraser and reranker over building a new embedding/vector RAG pipeline.

Rationale: Reusing the existing index avoided new infrastructure and embedding costs, while rephrasing/reranking handled multilingual quality gaps.

Org chart retrieval

Chose load the full org graph into context over keyword or vector retrieval over individual nodes.

Rationale: Relational hierarchy questions required seeing the whole graph, and the corpus was small enough for modern context windows.

Assistant platform strategy

Chose ChatGPT Enterprise with EU data residency over continuing Ada as the main internal assistant.

Rationale: Commercial data residency changed the build-vs-buy calculus and let the team focus GenAI expertise on user-facing use cases.

Gotchas (what bit them)

Self-hosting gave data control but worse operational economics.

Root cause: The Llama 2 deployment was complex, underperformed for chat, required large instances, and remained costly even with off-hours shutdowns.

Resolution: Move to Claude through AWS Bedrock after legal/security review confirmed data-handling guarantees.

Maximum privacy reduced observability.

Root cause: Ada stored no conversations, including for the development team, which made user-reported issue investigation harder.

Resolution: Accept the privacy tradeoff for the pilot, while recognizing the operational complexity it introduced.

Classic RAG was unnecessary or inferior for some sources.

Root cause: Backstage already had a search index, and the org chart required relational reasoning over a graph rather than isolated keyword matches.

Resolution: Use OpenSearch plus rephrasing/reranking for Backstage, and load the entire org graph into context for organizational questions.

Generic evaluation metrics missed real failures.

Root cause: Cosine similarity, answer relevance, and context relevance did not expose hallucinations, weak retrieval, table-structure loss, or multilingual issues.

Resolution: Build use-case-specific datasets and prioritize correctness plus correct-link retrieval metrics.

Rerankers improve relevance but add constraints.

Root cause: The reranker adds cost and has a much smaller token limit than Claude Sonnet, which restricts query size.

Resolution: Use reranking where its quality gains justify cost and query-length limits.

Rephrasers can reduce performance when applied blindly.

Root cause: A prompt-based rephraser hurt the Confluence assistant where retrieval already worked well.

Resolution: Treat rephrasers as targeted fixes for specific retrieval problems, not a default layer.

The custom assistant could not scale organizational demand.

Root cause: Leadership wanted every team to have an assistant, but a few MLEs could not keep pace while maintaining infrastructure.

Resolution: Phase out Ada and use ChatGPT Enterprise as the internal assistant base, preserving selected secret sauce through connectors and workflows.

Evaluation

Ada's evaluation evolved from synthetic Q&A datasets and generic metrics into use-case-specific datasets, multilingual test sets, user-feedback examples, LLM-as-judge correctness, correct-link retrieval scoring, LangSmith asynchronous batched evaluation, and weekly Airflow baselines. The team explicitly used evaluation to catch table chunking loss, weak retrieval, hallucinations, and retrieval-stage failures.

Metrics: cosine similarity · answer relevance · context relevance · correctness · correct links pulled · context relevance 0.63 to 0.73 · correct source link returned in 70% of cases · 120-example evaluation runtime · weekly baseline evaluation · query rephraser correctness lift

Outcomes

• Ada provided a private Claude-powered internal chatbot and safe GenAI learning environment
• the project expanded into specialized assistants for Confluence, Lumapps, Backstage, policy, and org-chart data
• Slack integration and summarization made Ada more accessible in employee workflows
• OpenSearch plus rephrasing and reranking matched traditional RAG performance for Backstage with less new infrastructure
• LangSmith reduced a 120-example evaluation run from 30 minutes to 3 minutes
• the team built practical expertise in RAG, indexing, evaluation, security governance, and EU legal compliance
• Ada was gradually phased out after ChatGPT Enterprise EU data residency made a commercial platform viable
• selected Ada capabilities were planned for migration through MCP connectors, Custom GPTs, and n8n workflows

Ada ran for about a year and a half, starting at the end of 2023 and phasing out after Q1 2025. The dedicated team was three MLEs, two software engineers, a PO, and a manager. Measured improvements included Backstage context relevance from 0.63 to 0.73, 70% correct source-link return, 120-example eval runtime reduced from 30 minutes to 3 minutes, and a 10% correctness lift from a simple moderation rephraser.