LLM-Powered Data Discovery and Documentation Platform

grab · Tech · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data discovery and documentation

Grab needed employees to find trustworthy datasets across a very large data lake without relying on keyword-only search, poor documentation coverage, or tribal Slack knowledge.

Constraints:

• The data lake contained more than 200,000 tables plus Kafka streams, production databases, and ML features.
• Existing Hubble/Datahub search was useful for reference lookup but weak for semantic discovery queries.
• Only 20% of frequently queried P80 tables had documentation before the project.
• Data consumers were asking colleagues in Slack and often taking multiple days to find the right dataset.

Architecture

Grab improved Hubble search, generated table documentation with GPT-4, published documented datasets into Glean, and built HubbleIQ as an LLM-based data-discovery bot available in Hubble UI and Slack. The architecture combines generated context, enterprise search, and channel integration so users can ask contextual discovery questions without leaving their workflow.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Hubble_catalog["Hubble metadata catalog"] --> Documentation_engine["Documentation generation engine"]
  Documentation_engine --> Producer_review["Data producer review"]
  Producer_review --> Glean_search["Glean enterprise search"]
  Hubble_catalog --> Glean_search
  Glean_search --> HubbleIQ["HubbleIQ bot"]
  HubbleIQ --> Hubble_UI["Hubble UI"]
  HubbleIQ --> Slack_channels["Slack ask channels"]

Components

Data flow

Hubble metadata and table schemas feed a GPT-4 documentation generation engine. Reviewed or AI-tagged documentation is indexed into Glean. HubbleIQ uses a custom system prompt and Glean access to answer semantic discovery questions, then appears above regular Hubble search results or inside Slack ask channels.

Why this architecture

Grab separated known-item search from semantic discovery: Elasticsearch enhancements handled exact and partial lookup, while HubbleIQ used generated documentation and Glean-backed LLM search for contextual questions that previously went to humans in Slack.

Models used: GPT-4 · Glean Apps LLM · LLMs

Scale

• Grab's discovery problem covered over 200,000 data-lake tables plus other data assets.
  “With over 200,000 tables in our data lake, along with numerous Kafka streams, production databases, and ML features, locating the most suitable dataset”article, stated
• Hubble's click-through rate rose from 82% to 94% after search improvements.
  “our monthly average click-through rate was only 82%, meaning that in 18% of sessions, users abandoned their searches without clicking on any dataset.”article, stated

  “As a result of these enhancements, the click-through rate rose steadily over the course of the half to 94%, a 12 percentage point increase.”article, stated
• Documentation coverage for critical P80 tables rose from 20% to about 90%.
  “Documentation coverage for our data lake tables was low, with only 20% of the most frequently queried tables (colloquially referred to as P80 tables) having existing documentation.”article, stated

  “As a result, documentation coverage for P80 tables increased by 70 percentage points to ~90%.”article, stated
• A follow-up survey found 73% of respondents found dataset discovery easy.
  “A follow-up survey revealed that 73% of respondents found it easy to discover datasets, marking a substantial 17 percentage point increase”article, stated

Alternatives considered (roads not taken)

Relying on vanilla Elasticsearch/Datahub search

Rejected because: It could handle keyword reference searches but could not use user context for semantic discovery.

Continuing Slack-based tribal-knowledge discovery

Rejected because: The process kept humans in the loop and caused multi-day discovery journeys.

Launching HubbleIQ without first improving documentation coverage

Rejected because: The team first built a context store because HubbleIQ needed high documentation coverage before LLM discovery could work.

Patterns

rag-retrieval (retrieval)

HubbleIQ answers semantic data-discovery questions by accessing documented Hubble datasets indexed in Glean.

human-in-the-loop-gate (safety)

AI-generated table documentation is visibly tagged until data producers accept or edit it.

eval-driven-development (evaluation)

The team tracked search click-through, documentation usefulness, survey outcomes, and planned an evaluator model for generated documentation quality.

Tradeoffs

Use Glean rather than build a bespoke enterprise search stack for HubbleIQ

Chose Glean Apps and Glean-indexed Hubble datasets over a custom semantic-search platform built from scratch.

Rationale: Grab explicitly chose Glean to speed up go-to-market while using documentation already made available from Hubble.

Expose generated documentation before full human finalization

Chose AI-generated tag plus producer review over waiting for all documentation to be manually written and approved.

Rationale: The tag made provenance visible while accelerating documentation coverage for critical tables.

Gotchas (what bit them)

Keyword catalog search could not satisfy contextual discovery queries.

Root cause: Elasticsearch search was tuned for known-dataset reference queries, not user-provided semantic context.

Resolution: Keep improving exact/partial search while adding HubbleIQ for semantic queries.

LLM discovery was limited by sparse documentation.

Root cause: Only a minority of important tables had useful documentation for the bot or users to reason over.

Resolution: Generate table documentation from schemas/sample data and ask producers to review it.

Discovery workflows were leaving the catalog UI and depending on colleagues.

Root cause: Users expected contextual questions to fail in Hubble and sent them to data producers over Slack.

Resolution: Integrate HubbleIQ into Slack ask channels as the first line of defence.

Evaluation

Grab measured search click-through, documentation coverage/usefulness, user survey changes, monthly active users, and planned evaluator-model checks for generated documentation quality.

Metrics: click-through rate rose to 94% · documentation coverage for P80 tables rose to about 90% · ~95% of users found generated docs useful · 73% found dataset discovery easy

Outcomes

• HubbleIQ added LLM-powered semantic discovery to Hubble UI and Slack.
• Documentation coverage for critical tables reached about 90%.
• Search click-through rose to 94% and monthly active users reached an all-time high.

The system targets Grab's large enterprise data lake with over 200,000 tables and many additional streams, databases, and ML features.