AI Agents for Documenting Tribal Knowledge in Large-Scale Data Pipelines

meta · Tech · 2026 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI coding assistants for large data pipelines

Meta’s agents lacked undocumented codebase knowledge across four repositories, three languages, and 4,100+ files.

Constraints:

• tribal knowledge
• cross-module dependencies
• stale references
• context size

Architecture

Meta built a pre-compute knowledge engine for AI coding assistants on a proprietary config-as-code data pipeline. A swarm of 50+ specialized agents maps repositories, analyzes modules, writes concise 25–35 line context files, runs multiple critic/fixer/upgrader/tester passes, builds routing/dependency lookups, and periodically refreshes/repairs the resulting knowledge layer. The generated context is model-agnostic, so future coding agents can navigate undocumented dependencies without re-discovering tribal knowledge every task.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  codebase-map["codebase-map"]
  explorer-agents["explorer-agents"]
  module-analysts["module-analysts"]
  context-writers["context-writers"]
  critic-agents["critic-agents"]
  fixers["fixers"]
  routing-upgraders["routing-upgraders"]
  prompt-testers["prompt-testers"]
  context-files["context-files"]
  maintenance-jobs["maintenance-jobs"]
  codebase-map --> explorer-agents
  explorer-agents --> module-analysts
  module-analysts --> context-writers
  context-writers --> critic-agents
  critic-agents --> fixers
  fixers --> context-files
  context-files --> routing-upgraders
  routing-upgraders --> prompt-testers
  context-files --> maintenance-jobs
  maintenance-jobs --> critic-agents

Components

Data flow

Explorer agents first map the codebase. Module analysts read each file and answer five questions about configuration, modification patterns, failure-causing non-obvious patterns, cross-module dependencies, and buried tribal knowledge. Writers turn those answers into concise context files; critic agents score and improve them; fixers correct issues; routing upgraders build lookup paths; prompt testers validate personas and queries; scheduled maintenance later checks paths, gaps, critics, and stale references.

"Two explorer agents mapped the codebase, 

• 11 module analysts read every file and answered five key questions, 
• Two writers generated context files, and 
• 10+ critic passes ran three rounds of independent quality review"

— article, stated "The five questions each analyst answered per module:

1. What does this module configure?
2. What are the common modification patterns?
3. What are the non-obvious patterns that cause build failures?
4. What are the cross-module dependencies?"

Why this architecture

The architecture front-loads context discovery because coding agents were compiling subtly wrong code when they lacked undocumented domain constraints. Meta rejects encyclopedia-sized context in favor of actionable compass files and quality gates: concise context reduces tool calls, while independent critics and freshness checks prevent stale or hallucinated guidance from becoming worse than no context.

Models used: large-context-window model

Scale

• The target pipeline spans four repositories, three languages, and over 4,100 files.
  “one of Meta’s large-scale data processing pipelines – spanning four repositories, three languages, and over 4,100 files”article, stated
• The pre-compute run used more than 50 specialized agents and produced 59 context files.
  “a swarm of 50+ specialized AI agents that systematically read every file and produced 59 concise context files encoding tribal knowledge that previously lived only in engineers’ heads.”article, stated
• AI navigation coverage rose from about 5% to 100% of code modules, and documented more than 50 non-obvious patterns.
  “AI agents now have structured navigation guides for 100% of our code modules (up from 5%, covering all 4,100+ files across three repositories). We also documented 50+ “non-obvious patterns,””article, stated
• Precomputed context reduced AI agent tool calls and tokens by roughly 40% per task in preliminary six-task tests.
  “In preliminary tests on six tasks against our pipeline, agents with pre-computed context used roughly 40% fewer tool calls and tokens per task.”article, stated
• Complex workflow guidance dropped from roughly two days of research/consultation to about 30 minutes.
  “Complex workflow guidance that previously required ~two days of research and consulting with engineers now completes in ~30 minutes.”article, stated

Alternatives considered (roads not taken)

Let coding agents explore the proprietary codebase on demand for every task

Rejected because: Agents burned many tool calls, missed naming/dependency patterns, and produced subtly wrong code.

Large exhaustive documentation files

Rejected because: The design principle was concise, actionable navigation rather than encyclopedia-like context.

Patterns

context-compaction (context-engineering)

Agents distilled thousands of files into 59 concise context files that act as navigation guides.

multi-agent-orchestration (documentation)

Specialized explorer, analyst, writer, critic, fixer, and upgrader agents divide the mapping work.

Tradeoffs

How much context to precompute.

Chose concise compass files over exhaustive encyclopedia documentation.

Rationale: The approach prioritizes navigation guidance that reduces tool calls without overloading the context window.

How much documentation context to generate for coding agents.

Chose Concise 25–35 line compass files with routing and quality gates. over Exhaustive encyclopedia-style context files or no precomputed context..

Rationale: Meta argues actionable navigation beats exhaustive docs and that absent context is measurably more expensive due to exploration and subtle mistakes.

Gotchas (what bit them)

Agents produced plausible but subtly wrong code without undocumented context.

Root cause: The pipeline’s config-as-code design spans registries, routing, DAGs, validation, generation, and automation scripts.

Resolution: Teach agents the codebase before exploration by precomputing a maintained context layer.

Precomputed context can become stale as code reviews and commits create new tribal knowledge.

Root cause: A static context layer decays unless it is refreshed against file paths, coverage gaps, and emerging patterns.

Resolution: Run automated refresh and validation jobs that detect stale references and newly forming patterns.

Evaluation

Meta measured coverage, documented patterns, and tool-call reduction.

Metrics: 100% module coverage · 50+ non-obvious patterns · 40% fewer tool calls

Outcomes

• The context layer increased module coverage from 5% to 100%.

Internal Meta pipeline case across 4,100+ files.