Building a Production RAG-Based Slackbot for Developer Support

vespa · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Community developer support RAG Slackbot

Vespa needed to absorb a surge of community support questions by turning historical Slack conversations and documentation into a searchable, privacy-aware RAG support bot that could answer questions in Slack and learn from new conversations.

Constraints:

• Docker pulls and community interest rose quickly, creating more Slack questions
• The bot had to answer from both previous conversations and Vespa documentation
• Users needed to consent before questions were sent to OpenAI
• Users needed control over whether their messages were indexed
• Indexed Slack messages had to be anonymized
• The deployment required production HTTP server mode and infrastructure management

Architecture

Vespa interns built a Kotlin Slackbot that listens for mentions and messages, stores consented Slack messages in a Vespa application, groups retrieved messages by Slack thread, ranks them with hybrid semantic/BM25 ranking plus emoji-feedback signals, and sends selected context to OpenAI for summarization. The bot extends Vespa's existing documentation-search RAG backend and is deployed on GCP with Terraform managed through SpaceLift.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  slack-events[Slack Events] --> slackbot[Slackbot]
  slackbot --> vespa-app[Vespa App]
  vespa-app --> slack-message-schema[Slack Message Schema]
  slack-message-schema --> text-embedding-field[Text Embedding Field]
  text-embedding-field --> hybrid-rank-profile[Hybrid Rank Profile]
  feedback-signal[Feedback Signal] --> hybrid-rank-profile
  hybrid-rank-profile --> documentation-search-backend[Documentation Search Backend]
  documentation-search-backend --> openai-summarizer[OpenAI Summarizer]
  openai-summarizer --> slackbot
  gcp-deployment[GCP Deployment] --> slackbot
  terraform[Terraform] --> gcp-deployment
  spacelift[SpaceLift] --> terraform

Components

Data flow

When a user mentions the bot, the AppMentionEvent handler queries the Vespa app for matching conversations and produces an answer. When users consent to indexing, the MessageEvent handler stores message text with message and thread IDs. Vespa embeds message text into a tensor field, ranks results with a hybrid rank profile, groups hits by thread, biases ranking with reaction feedback, and sends retrieved context to OpenAI for summarization.

Why this architecture

Vespa reused its existing documentation-search RAG backend rather than building an isolated bot, because the community Slack corpus complemented official docs. Slack thread IDs preserved conversational context, hybrid ranking balanced semantic and lexical matching, emoji reactions gave lightweight relevance feedback, and consent/anonymization controls made production ingestion of community messages acceptable.

Models used: OpenAI · Vespa embedder · 384-dimensional text embeddings

Scale

• Vespa Docker pulls rose from 2M to 11M in a few months.
  “Vespa.ai saw a dramatic rise in interest in early September 2023, with pulls from vespaengine/vespa soaring from 2M to 11M in just a few months. This surge led to a flood of questions on their Slack channel.”article, stated
• The Slack message embedding field is a 384-dimensional float tensor.
  “field text_embedding type tensor(x[384]) { indexing: input text | embed embedder | attribute | index attribute { distance-metric: angular }”article, stated
• The hybrid rank profile weights semantic similarity at 0.7 and scaled BM25 at 0.3.
  “expression: scale(bm25(text)) } first-phase { expression: 0.7 * semantic_text + 0.3 * scaled_bm25_text rank-score-drop-limit: 0.0 } } }”article, stated

Alternatives considered (roads not taken)

Build a standalone Slackbot search backend

Rejected because: The existing documentation search backend already provided RAG over docs, samples, and blog posts, so the Slackbot could extend it.

Run the bot in Socket Mode for production

Rejected because: Socket Mode was recommended for development, while HTTP server mode was recommended for deployment.

Use Terraform without additional IaC management

Rejected because: Terraform state and configuration were difficult for the interns, while SpaceLift smoothed state handling and integration.

Patterns

rag-retrieval (knowledge-access)

The bot retrieves historic Slack conversations and documentation paragraphs before summarizing an answer with OpenAI.

hybrid-search (retrieval-ranking)

Vespa combines semantic text similarity and scaled BM25 in its hybrid2 rank profile.

data-quality-loop (feedback)

Thumbs-up and thumbs-down reactions alter ranking toward helpful historic responses.

privacy-preserving-processing (data-access)

The Slackbot requires consent for OpenAI sharing, lets users opt into indexing, and anonymizes users.

retrieval-ranking (search)

Retrieved messages are ranked individually and then grouped by thread using the maximum relevance score.

Tradeoffs

Extend the existing documentation-search backend

Chose add Slack schemas and searchers to search.vespa.ai infrastructure over build a separate RAG application from scratch.

Rationale: The existing backend already summarized documentation with OpenAI, so Slack conversations became an additional knowledge layer.

Blend semantic and lexical ranking

Chose 0.7 semantic similarity and 0.3 scaled BM25 over pure vector search or pure BM25.

Rationale: The rank profile emphasizes conceptual similarity while preserving exact lexical relevance signals.

Use HTTP server mode for deployment

Chose public endpoint HTTP server mode over Socket Mode.

Rationale: Socket Mode simplified development, while HTTP server mode is the recommended deployment mode.

Gotchas (what bit them)

Community Slack content cannot be indexed casually.

Root cause: The bot sends questions to OpenAI and stores user messages, creating privacy and consent obligations.

Resolution: Vespa added explicit consent, opt-in indexing, anonymization, and reaction-based feedback controls.

Slack conversations need thread-aware retrieval.

Root cause: Individual Slack messages are fragments of conversations, so relevance at message level is not enough.

Resolution: The schema stores thread IDs and groups hits by thread.

Infrastructure-as-code slowed production deployment.

Root cause: The interns had no prior Terraform experience and ran into configuration and state-management friction.

Resolution: They used SpaceLift, already used by Vespa, to manage Terraform state and integration.

Evaluation

Vespa evaluated the bot through deployment-readiness requirements, user feedback reactions, continuous indexing quality, and whether the bot could improve answers as more consented Slack interactions were processed.

Metrics: Docker pulls · Slack question volume · thumbs-up reactions · thumbs-down reactions · retrieval relevance score · rank score drop limit · indexed messages

Outcomes

• Vespa built a Slackbot that answers community questions from previous Slack conversations and documentation.
• The bot indexes consented Slack messages into Vespa and retrieves thread-grouped context.
• The ranking profile combines embeddings, BM25, and emoji feedback.
• The bot includes consent, anonymization, feedback, slash-command, and deployment machinery.
• The interns deployed with GCP, Terraform, and SpaceLift.

The bot responded to a community growth moment where Docker pulls rose from 2M to 11M, and it used a 384-dimensional embedding field with a hybrid rank profile weighting semantic similarity 70% and BM25 30%.