Productionizing LLM-Powered Data Governance with LangChain and LangSmith

grab · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data governance metadata and PII classification

Grab needed to automate governance metadata generation and data-entity classification across a large data lake while reducing dependence on third-party classifiers, managing LLM capacity limits, and preserving human verification for sensitive privacy decisions.

Constraints:

• Classification decisions affect data sensitivity tiers and privacy rules.
• The original prompt asked the model to distinguish 21 tags, including many non-PII tags that distracted from PII detection.
• Large tables with more than 150 columns exceeded the model's attention capacity.
• Data owners still need to verify classifications to prevent misclassification in high-stakes governance workflows.

Architecture

Metasense V2 decomposes Grab's data-governance classification task into PII and non-PII subtasks, reduces tags from 21 to 8 for the PII model, shortens prompts from 1,254 to 737 words, chunks tables with more than 150 columns, and uses LangChain/LangSmith to manage prompt versions, datasets, evaluations, collaboration, and deployment configuration. Human data owners verify outputs, while monitoring alerts and improvement protocols watch for misclassification drift.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  data-lake-tables["Data lake tables"] --> table-chunker["Table chunker"]
  data-lake-tables["Data lake tables"] --> metasense-classifier["Metasense classifier"]
  table-chunker["Table chunker"] --> pii-classifier["PII classifier"]
  table-chunker["Table chunker"] --> non-pii-classifier["Non-PII classifier"]
  pii-classifier["PII classifier"] --> langchain-backend["LangChain backend"]
  non-pii-classifier["Non-PII classifier"] --> langchain-backend["LangChain backend"]
  pii-classifier["PII classifier"] --> metasense-classifier["Metasense classifier"]
  non-pii-classifier["Non-PII classifier"] --> metasense-classifier["Metasense classifier"]
  langchain-backend["LangChain backend"] --> langsmith-workspace["LangSmith workspace"]
  langsmith-workspace["LangSmith workspace"] --> evaluation-datasets["Evaluation datasets"]
  metasense-classifier["Metasense classifier"] --> governance-rules["Governance rules"]
  governance-rules["Governance rules"] --> data-owner-verification["Data owner verification"]
  langsmith-workspace["LangSmith workspace"] --> monitoring-alerts["Monitoring alerts"]
  monitoring-alerts["Monitoring alerts"] --> data-owner-verification["Data owner verification"]

Components

Data flow

Data lake tables and columns enter Grab's metadata generation workflow. Metasense chunks oversized tables, sends PII and non-PII classification subtasks through LangChain-backed prompts, and records results as governance tags that combine with privacy rules to determine sensitivity tiers. LangSmith stores prompts and evaluation datasets, lets data scientists iterate prompt versions in commit mode, exposes accuracy/latency/error metrics, and supports deployment by swapping prompt identifiers in service configuration. Data owners verify outputs and monitoring alerts trigger improvement protocols when misclassification thresholds are crossed.

Why this architecture

Grab's architecture manages LLM capacity by reducing task complexity before asking the model to classify data. Splitting PII from non-PII, shortening prompts, reducing tag space, and chunking wide tables all make the classification problem smaller. LangSmith gives the cross-functional team a shared prompt/eval/deployment workflow, while human verification and alerts acknowledge that governance errors still carry risk.

Models used: LLM classifier · LangChain · LangSmith

Scale

• Metasense V1 initially scanned more than 20,000 data entries.
  “initially scanned more than 20,000 data entries at an average rate of 300-400 entities per day.”summary, stated
• The initial scan rate was 300-400 entities per day.
  “initially scanned more than 20,000 data entries at an average rate of 300-400 entities per day.”summary, stated
• The original prompt covered 21 tags.
  “The original prompt required the model to distinguish between 21 tags”summary, stated
• The PII classifier reduced tags from 21 to 8.
  “Tag reduction: Reduced the number of tags from 21 to 8”summary, stated
• The prompt was shortened from 1,254 to 737 words.
  “Reduced the word count in the prompt from 1,254 to 737 words”summary, stated
• Tables with more than 150 columns are chunked.
  “Tables with more than 150 columns are split into smaller tables”summary, stated

Alternatives considered (roads not taken)

Third-party data classification services

Rejected because: They restricted customization and required significant resources for custom model training.

One model prompt for all 21 tags

Rejected because: Non-PII tags distracted from the primary PII task and overloaded model capacity.

Classify very wide tables in one request

Rejected because: LLM attention/capacity degraded when too many columns were classified simultaneously.

Fully automated governance without human verification

Rejected because: High-stakes data classification still needs human oversight to prevent misclassification.

Patterns

workflow-decomposition (architecture)

Grab splits one broad governance classifier into PII and non-PII subtasks and chunks wide tables.

eval-driven-development (quality)

Table-owner feedback and Data Governance Office classifications become datasets for model and prompt improvement in LangSmith.

observability-tracing (operations)

LangSmith provides prompt versioning, metrics, datasets, and evaluation visibility for the LLM workflow.

human-in-the-loop-gate (governance)

Data owners verify classifications to protect against high-impact governance misclassification.

safety-policy-enforcement (privacy)

The classifier feeds privacy rules and sensitivity tiers, with alarms when misclassification rates cross thresholds.

Tradeoffs

Use LLM prompt engineering instead of custom classifier training

Chose LLM-based classification with prompt/task design over Third-party services or resource-heavy custom model training.

Rationale: LLMs offered a more flexible and affordable path for enterprise-scale governance classification.

Shorten and simplify prompts

Chose Concise 737-word prompt over More detailed 1,254-word prompt.

Rationale: Removing unnecessary detail helped model capacity focus on the classification task.

Adopt LangChain and LangSmith

Chose Third-party LLMOps collaboration and prompt tooling over Pure custom prompt-management workflow.

Rationale: The tools accelerated experimentation and cross-functional collaboration, with some dependency tradeoff.

Gotchas (what bit them)

Too many classification tags reduce model focus.

Root cause: The model had to discriminate 21 tags, including many non-PII categories unrelated to the primary risk.

Resolution: Split PII/non-PII classification and reduce PII tags to 8.

Wide tables overload LLM attention.

Root cause: Classifying too many columns at once leaves insufficient capacity to reason about each column.

Resolution: Split tables with more than 150 columns into smaller chunks.

PII can hide in mostly non-PII fields.

Root cause: Business emails, nested JSON, and passenger communications can include legal names, phone numbers, or email addresses amid other metadata.

Resolution: Use feedback-derived edge cases and simplified PII-focused prompts to improve detection.

Evaluation

Grab evaluates with LangSmith datasets and metrics across accuracy, latency, and error rate; monitors misclassification rates periodically; triggers alarms on thresholds; and uses human owner verification for governance-critical outputs.

Metrics: accuracy · latency · error rate · misclassification rate · threshold alarms · data-owner verification

Outcomes

• Metasense V2 achieved low misclassification rates according to the case study, though exact rates are not provided.
• Coverage grew to the vast majority of Grab's data lake tables.
• Manual workload for data owners was significantly reduced but not eliminated.
• LangChain/LangSmith improved experimentation, collaboration, and prompt deployment workflow.

Concrete anchors include 20,000+ entries, 300-400 entities/day, 21 to 8 tag reduction, 1,254 to 737 prompt-word reduction, and 150-column chunking threshold.