LLM-Powered Data Classification System for Enterprise-Scale Metadata Generation

grab · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data governance and metadata classification

Grab needed to classify petabyte-scale data entities for sensitive information and PII so access controls, discovery, ABAC, and masking could operate at finer granularity than manual schema-level tagging allowed.

Constraints:

• Grab manages petabyte-level data across database tables, Kafka schemas, and data lake entities.
• Manual campaigns did not scale to the volume, velocity, and variety of table-level classification.
• Building custom classifiers would require a dedicated data science team, governance-rule expertise, and labeled training data.
• LLM outputs had to be structured enough for downstream data platforms and cheap enough for production load.

Architecture

Grab integrated GPT-3.5-style LLM classification into Gemini, its existing data-classification orchestration service. Data platforms submit classification requests, Gemini groups them into mini-batches through a message queue, rate-limits provider calls, prompts the LLM as a column tag generator with few-shot examples and a default None tag, publishes predictions to Kafka, and asks users to verify tags weekly so prompts can improve and downstream ABAC and Dynamic Data Masking can consume the classifications.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Data_platforms["Data platforms"] --> Gemini["Gemini orchestration service"]
  Gemini --> Message_queue["Message queue batching"]
  Message_queue --> Rate_limiter["Workflow rate limiter"]
  Rate_limiter --> Cloud_APIs["Cloud provider APIs"]
  Cloud_APIs --> LLM_tags["LLM column tag generator"]
  Prompt_controls["Prompt engineering controls"] --> LLM_tags
  LLM_tags --> Kafka_queue["Kafka prediction queue"]
  Kafka_queue --> User_verification["User verification workflow"]
  User_verification --> Prompt_analytics["Prompt analytics pipeline"]
  Kafka_queue --> Data_platforms
  Data_platforms --> ABAC_masking["ABAC and masking consumers"]

Components

Data flow

A data platform sends data-entity classification requests to Gemini. Gemini schedules and groups requests into mini-batches, uses a workflow-level rate limiter to call cloud LLM APIs, prompts the model to assign column tags in a fixed output format, publishes predictions to Kafka, and downstream platforms ask users to verify tags weekly for correctness and prompt iteration.

Why this architecture

Grab used an orchestration service around the LLM because classification had to run at data-platform scale, preserve structured downstream outputs, avoid provider throttling, and stay operable by governance teams through natural-language prompts instead of custom model-training cycles.

Models used: GPT-3.5 · LLM column tag generator · few-shot prompting

Scale

• Grab manages petabyte-level data across countless entities.
  “At Grab, we deal with PetaByte-level data and manage countless data entities ranging from database tables to Kafka message schemas.”article, stated
• The system scanned more than 20,000 data entities within one month of rollout.
  “Within a month since its rollout, we have scanned more than 20,000 data entities, averaging around 300-400 entities per day.”article, stated
• The system averages around 300-400 classified entities per day.
  “averaging around 300-400 entities per day.”article, stated
• Grab estimated the automated process saves roughly 360 man-days per year.
  “we are saving approximately 360 man-days per year for the company.”article, stated
• 80% of surveyed data owners reported the new tagging process helped them.
  “during an internal survey done among data owners at Grab in September 2023, 80% reported that this new tagging process helped them”article, stated

Alternatives considered (roads not taken)

Manual schema/table classification campaigns

Rejected because: Manual work could not keep up with the volume, velocity, and variety of table-level data classification.

In-house custom classifiers

Rejected because: Custom models would require a dedicated data science team, governance-rule learning, and manually labeled training data.

Free-text LLM output

Rejected because: Downstream workflows needed fixed-format tags rather than unconstrained prose.

Always requiring user verification

Rejected because: Grab plans to reduce mandatory verification once accuracy and confidence become high enough.

Patterns

tool-use-orchestration (workflow)

Gemini orchestrates batching, rate limiting, cloud LLM calls, Kafka publishing, and downstream user verification around the classifier.

structured-output-contract (output control)

Prompt engineering constrains the LLM to return fixed-format column tags suitable for downstream processing.

prompt-chaining (prompting)

The prompt uses task instructions, examples, and a None fallback to improve classification behavior without training a custom model.

human-in-the-loop-gate (quality)

Users verify classified tags weekly, improving correctness and enabling iterative prompt improvement before verification can be relaxed.

eval-driven-development (operations)

Grab is building analytical pipelines to measure prompt-version effectiveness and iterate prompts faster.

Tradeoffs

Use LLM classification instead of building custom classifiers

Chose natural-language prompt-driven classification over dedicated in-house classifier training.

Rationale: Governance personnel could express requirements through prompts without a full ML training and labeling program.

Keep users in the verification loop initially

Chose weekly tag verification by respective users over immediate fully automated trust in predictions.

Rationale: Predictions are unverified model outputs until users validate correctness and provide feedback for prompt improvement.

Use a default None output for uncertainty

Chose explicit confusion fallback over forcing a sensitivity tag for every column.

Rationale: A default tag reduces overconfident classifications when the model cannot decide.

Build on the existing Gemini orchestration service

Chose integrate LLM classification into current data-classification orchestration over standalone LLM calls outside the governance workflow.

Rationale: Gemini already handled platform communication, scheduling, grouping, and downstream integration needs.

Gotchas (what bit them)

Schema-level sensitivity labels over-restricted non-sensitive data.

Root cause: A single Tier 1 table caused an entire schema with hundreds of tables to inherit strict access controls.

Resolution: Move toward finer-grained table/entity classification powered by automated LLM tagging.

Manual classification did not scale with Grab's data growth.

Root cause: The volume, velocity, and variety of data increased the cost of table-level classification beyond manual campaigns.

Resolution: Use a programmatic LLM classification service with batching and rate limiting.

Unstructured LLM output would break downstream workflows.

Root cause: Language models typically emit free text, while data platforms need predictable tags.

Resolution: Use prompt engineering, few-shot examples, and a fixed output format with a default None tag.

Cloud provider LLM calls can be throttled under batch classification load.

Root cause: Classification requests arrive from data platforms and are grouped into service-provider API calls.

Resolution: Attach a rate limiter at workflow level to call cloud provider APIs at their respective rates.

Evaluation

Grab evaluated the rollout using user tag-change behavior, an internal survey, throughput after rollout, and planned prompt-version analytics.

Metrics: average user tag changes · 80% survey helpfulness · 20,000+ entities scanned in one month · 300-400 entities per day · prompt-version metrics pipeline

Outcomes

• The system scanned more than 20,000 data entities within a month.
• Grab estimates about 360 man-days per year in tagging time savings.
• Classified tags now support ABAC and Dynamic Data Masking downstream.

The production load covers petabyte-level data and hundreds of daily entity classifications at current rollout scale.