RAG-Powered Terraform Support Slackbot

benchling · Tech · 2024 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Internal infrastructure support and Terraform Cloud knowledge access

Benchling engineers needed a faster way to answer Terraform Cloud questions without manually searching a 20-page FAQ and scattered Slack threads while managing a large multi-datacenter Terraform estate.

Constraints:

• The source cache for the original article is unavailable, so this extraction is based on the ZenML summary only.
• The knowledge base needed to combine Confluence documentation, public Hashicorp docs, and historical Slack threads.
• The prototype needed low operational overhead and used AWS managed services.
• The current implementation had limits around image processing, manual data synchronization, stateless conversations, and missing citation links.

Architecture

Benchling built a Terraform support Slackbot using a Slack App, API Gateway, a Python AWS Lambda, Amazon Bedrock knowledge bases, Claude 3.5 Sonnet v2 for response generation, Titan Text Embeddings v2 for embeddings, and OpenSearch Serverless for vector storage. Documentation from Confluence, Hashicorp public docs, and manually copied Slack threads is stored in S3 and ingested into the Bedrock knowledge base. The Lambda handles Slack requests, retrieves credentials from AWS Secrets Manager, calls Bedrock RetrieveAndGenerate/Retrieve APIs, and returns answers in Slack.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Slack App"] --> B["AWS API Gateway"]
  B --> C["AWS Lambda"]
  C --> D["AWS Secrets Manager"]
  C --> E["Amazon Bedrock"]
  E --> F["Titan Text Embeddings v2"]
  F --> H["OpenSearch Serverless"]
  E --> G["Claude 3.5 Sonnet v2"]
  I["S3 knowledge sources"] --> E
  E --> H
  C --> J["CloudWatch logs"]

Components

Data flow

Terraform support documents and selected Slack-thread histories are stored for Bedrock ingestion, embedded with Titan Text Embeddings v2, and indexed in OpenSearch Serverless. A Slack user asks a Terraform Cloud question, API Gateway forwards the webhook to Lambda, Lambda retrieves secrets and calls Bedrock RetrieveAndGenerate or Retrieve, Claude 3.5 Sonnet v2 generates the response over retrieved context, and the answer returns to Slack.

Why this architecture

Benchling chose a small managed-services RAG architecture because the problem was scoped internal knowledge lookup, not broad infrastructure automation. AWS Bedrock reduced infrastructure setup, Slack met engineers where they already worked, and a tightly scoped Terraform dataset reduced hallucination risk while making the prototype reusable for future internal help bots.

Models used: Claude 3.5 Sonnet v2 · Amazon Titan Text Embeddings v2 · Amazon Bedrock Knowledge Bases

Scale

• Benchling managed approximately 160,000 Terraform resources across five data centers.
  “managing approximately 160,000 Terraform resources across five data centers using a self-hosted Terraform Cloud implementation.”summary, stated
• Around 50 engineers made infrastructure changes monthly.
  “With around 50 engineers—ranging from infrastructure specialists to application engineers new to Terraform—making infrastructure changes monthly”summary, stated
• The Lambda function was configured with 128MB memory and a 60-second timeout.
  “The Lambda function is configured with 128MB of memory and a 60-second timeout, running on ARM64 architecture.”summary, stated
• CloudWatch logs were retained for 90 days.
  “CloudWatch logs are retained for 90 days, providing observability into the system's behavior.”summary, stated

Alternatives considered (roads not taken)

Continue manual search across a FAQ and Slack threads

Rejected because: Knowledge was scattered and difficult for engineers to find quickly.

Use Bedrock default 300-token chunking

Rejected because: FAQ answers spanned multiple paragraphs, so default chunks cut off answers midstream.

Manage Bedrock knowledge base infrastructure entirely in Terraform

Rejected because: The AWS Terraform provider did not yet support the Bedrock knowledge base resources.

Patterns

rag-retrieval (retrieval)

The Slackbot retrieves Terraform support context from a Bedrock knowledge base before Claude generates answers.

retrieval-ranking (retrieval)

OpenSearch Serverless stores vector embeddings used by Bedrock retrieval.

observability-tracing (operations)

CloudWatch logging gives the team runtime visibility into Lambda behavior.

safety-policy-enforcement (risk)

The team performed a security and privacy risk assessment before development.

Tradeoffs

Use AWS managed RAG services for a prototype.

Chose Bedrock knowledge bases, OpenSearch Serverless, API Gateway, and Lambda over Building and operating a custom RAG platform.

Rationale: The managed wizard reduced what would have been days of setup to minutes.

Tune chunking to answer shape instead of accepting defaults.

Chose Hierarchical chunking with 1500-token parents over Bedrock's 300-token default chunks.

Rationale: Long ordered FAQ answers needed enough context to stay intact, without including irrelevant material.

Use Slack as the support interface.

Chose A Slackbot over A separate knowledge-base portal.

Rationale: The problem was engineers asking internal support questions, and Slack already contained both the interaction channel and historical support threads.

Gotchas (what bit them)

Default chunking cut off multi-paragraph FAQ answers.

Root cause: Bedrock's 300-token default chunks were shorter than the team's ordered-step answers.

Resolution: Use hierarchical chunking with a 1500-token parent size, near the upper limit of the longest answers.

Important documentation context was visual and therefore unavailable to the knowledge base.

Root cause: The knowledge base could not process images in queries or include documentation images in responses.

Resolution: Record image processing as a limitation and future production maturity requirement.

Manual synchronization created freshness debt.

Root cause: Data sources required manual sync operations in the prototype.

Resolution: Plan CloudWatch event cron triggers for at least weekly automated synchronization.

Slack responses lacked visible citations.

Root cause: Bedrock UI citations were not passed through to Slack.

Resolution: Add citation links to Slack responses as a future improvement.

Evaluation

The summary reports chunking experiments, a security/privacy risk assessment, and prototype limitations; it does not provide a quantitative production evaluation beyond the operational configuration and lessons learned.

Metrics: Chunking quality · Security/privacy risk assessment · CloudWatch logs retained for 90 days · Manual limitation inventory

Outcomes

• The prototype demonstrated rapid deployment of an LLM-powered internal support tool.
• The Bedrock wizard reduced setup from a multi-day project to minutes.
• The team identified concrete production gaps before broad rollout.
• The implementation became a reference pattern for targeted internal help bots.

The summary reports approximately 160,000 Terraform resources, five data centers, around 50 monthly infrastructure-changing engineers, a 20-page FAQ, 128MB Lambda memory, 60-second timeout, and 90-day log retention.