AI-Driven Documentation Generation for dbt Data Models

loblaw_digital · E-commerce · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Analytics engineering documentation

Loblaw Digital needed dbt model and column documentation to stay complete and useful across thousands of data models without forcing engineers to manually maintain repetitive descriptions.

Constraints:

• thousands of dbt models
• business-user readability
• data security
• manual documentation debt

Architecture

Loblaw Digital used dbt-documentor as a metadata-only documentation pipeline: it ingests dbt manifest schema information, calls Vertex AI to draft model and column descriptions, and publishes the result into dbt documentation without reading the underlying data.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  dbt-manifest-metadata["dbt manifest metadata"]
  dbt-documentor["dbt-documentor"]
  vertex-ai-generator["Vertex AI generator"]
  security-boundary["Security boundary"]
  dbt-documentation-site["dbt documentation site"]
  dbt-manifest-metadata --> dbt-documentor
  dbt-documentor --> vertex-ai-generator
  security-boundary --> dbt-manifest-metadata
  security-boundary --> vertex-ai-generator
  dbt-documentor --> dbt-documentation-site

Components

Data flow

The documentation pipeline starts from dbt manifest metadata, sends schema structure to an LLM, and returns generated model and column descriptions into the dbt documentation surface.

Why this architecture

The metadata-only design addresses documentation debt while keeping raw business data out of the LLM path.

Models used: Vertex AI LLMs

Scale

• The workflow targets thousands of models across lines of business where manual descriptions were frequently omitted.
  “This information needs to be manually written down when developing each model, and with thousands of models across different lines of businesses, the description section of the documentation is often omitted.”article, stated

Alternatives considered (roads not taken)

Manual documentation across dbt files and metadata

Rejected because: The team automated relation-aware documentation because manual cross-referencing was cumbersome and error-prone.

Sending actual table data to the LLM

Rejected because: Loblaw kept the input to schema and manifest metadata so data integrity and security remained intact.

Leaving descriptions optional for each model author

Rejected because: The source says description sections were often omitted across thousands of models, so the system automates the repetitive documentation path.

Patterns

structured-output-contract (data-documentation)

The LLM writes structured model and column descriptions from dbt schema metadata rather than free-form ad hoc documentation.

privacy-preserving-processing (data-governance)

The documentation generator uses schema and relation metadata without reading the underlying business data.

Tradeoffs

What data to expose to the documentation LLM.

Chose dbt manifest and relation metadata over actual production data.

Rationale: Metadata gives enough structure to generate useful descriptions while avoiding unnecessary data exposure.

Gotchas (what bit them)

Documentation debt made dbt models confusing for business users.

Root cause: Manual model and column descriptions were often omitted as the number of models grew across lines of business.

Resolution: Generate descriptions from compiled dbt metadata so documentation coverage can scale with the model graph.

Evaluation

The post demonstrates generated dbt documentation and focuses on workflow efficiency rather than a formal quantitative benchmark.

Metrics: documentation coverage · manual cross-reference reduction · business-user clarity

Outcomes

• The workflow automates dbt model documentation generation while preserving data-security boundaries.
• Business users get clearer model and column descriptions in the dbt documentation site.

Non-agentic LLM automation case; valuable for pattern extraction but not an autonomous agent deployment.