Building a Custom LLM for Automated Documentation Generation

databricks · Tech · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Data-catalog documentation generation

Databricks needed to generate high-quality table and column documentation for Unity Catalog at customer scale, but off-the-shelf SaaS LLMs created quality drift, throughput limits, and cost problems.

Constraints:

many enterprise tables are undocumented
the system needs to serve tens of thousands of organizations and potentially millions of tables per organization
SaaS LLM quality can change without Databricks code changes
API quota and inference cost constrain full-catalog documentation
training must avoid customer data

Architecture

Databricks moved from a hackathon prototype using SaaS LLMs to a bespoke fine-tuned MPT-7B model for Unity Catalog documentation. It synthesized about 3,600 training examples from NAICS and internal taxonomy data, evaluated candidate outputs with double-blind human comparison on 62 unseen tables, registered production models in Unity Catalog, distributed them with Delta Sharing, and served them through optimized LLM serving across regions.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  unity-catalog-metadata[Unity Catalog metadata] --> documentation-suggestion-ui[Documentation suggestion UI]
  synthetic-training-data-generator[Synthetic training data generator] --> fine-tuning-pipeline[Fine-tuning pipeline]
  fine-tuning-pipeline --> evaluation-framework[Evaluation framework]
  fine-tuning-pipeline --> model-registry[Model registry]
  model-registry --> delta-sharing-distribution[Delta Sharing distribution]
  delta-sharing-distribution --> optimized-llm-serving[Optimized LLM serving]
  optimized-llm-serving --> fine-tuned-documentation-model[Fine-tuned documentation model]
  fine-tuned-documentation-model --> unity-catalog-metadata[Unity Catalog metadata]
  fine-tuned-documentation-model --> documentation-suggestion-ui[Documentation suggestion UI]
  documentation-suggestion-ui --> production-metrics[Production metrics]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Unity Catalog metadata`	Tables, schemas, and columns that need documentation suggestions.	`Documentation suggestion UI`, `Fine-tuned documentation model`
`Documentation suggestion UI`	User workflow for accepting, bulk accepting, or modifying generated table and column documentation.	`Unity Catalog metadata`, `Fine-tuned documentation model`, `Production metrics`
`Synthetic training data generator`	Builds table schemas and target documentation from NAICS and Databricks internal taxonomy sources.	`Fine-tuning pipeline`
`Fine-tuning pipeline`	Fine-tunes candidate 7B models on the generated documentation dataset.	`Synthetic training data generator`, `Evaluation framework`, `Model registry`
`Evaluation framework`	Double-blind human comparison of candidate model outputs and evaluator agreement.	`Fine-tuning pipeline`
`Model registry`	Unity Catalog model registration, governance, and lineage for production models.	`Fine-tuning pipeline`, `Delta Sharing distribution`
`Delta Sharing distribution`	Distributes selected models to production regions around the world.	`Model registry`, `Optimized LLM serving`
`Optimized LLM serving`	Provisioned, optimized serving layer for low-latency, high-throughput documentation inference.	`Delta Sharing distribution`, `Fine-tuned documentation model`
`Fine-tuned documentation model`	MPT-7B model specialized for table descriptions and column comments.	`Optimized LLM serving`, `Unity Catalog metadata`, `Documentation suggestion UI`
`Production metrics`	Acceptance-rate and real-world usage metrics used to monitor model quality after launch.	`Documentation suggestion UI`

Data flow

A table schema enters the documentation workflow. Training examples were synthesized from public and internal taxonomy data, candidate models were fine-tuned and double-blind evaluated, the selected MPT-7B model was registered in Unity Catalog, Delta Sharing distributed it to production regions, optimized serving generated table and column suggestions, and users accepted, bulk accepted, or edited them.

3 supporting quotes

“This new workflow would automatically suggest descriptions for the tables and columns and allow users to either individually accept, bulk accept, or modify the suggestions”article, stated

“Then we synthesized CREATE TABLE statements using the above use cases to yield a diverse set of tables and generated sample responses including table descriptions and column comments”article, stated

“Once the models are registered in UC, they can be served using the new optimized LLM serving, which provides significant performance improvement in terms of throughput and latency improvement compared to traditional serving for LLM serving.”article, stated

Why this architecture

Databricks treated documentation as a narrow applied-ML task rather than a general chatbot problem. A smaller fine-tuned model gave control over quality, fit A10 GPUs for horizontal throughput, reduced prompt size and inference cost, and let Databricks govern model lineage in Unity Catalog while distributing models globally.

3 supporting quotes

“* The smaller 7B models, after fine-tuning, were already meeting the quality bar. It was significantly better than the cheaper version of the SaaS model, and roughly equivalent to the more expensive version.”article, stated

“* The smaller model could fit comfortably and be served using A10 GPUs, which were more readily available. Their abundance would mean higher inference throughput for the task.”article, stated

“Unity Catalog: The models that we use in production are registered in Unity Catalog (UC), providing the governance we need”article, stated

Models used: MPT-7B · Llama2-7B · MPT-30B · Llama-2-13B · SaaS LLMs

Scale

More than 80 percent of Databricks table metadata updates are AI-assisted.
“We have been humbled by the reception of this feature among our customers. Today, more than 80% of the table metadata updates on Databricks are AI-assisted.”article, stated
The target customer scale includes tens of thousands of organizations and single organizations with millions of tables.
“We work with tens of thousands of organizations, and it is not uncommon that a single organization would have millions of tables.”article, stated
The bespoke model took two engineers one month and less than $1,000 in compute cost to develop.
“The new model took 2 engineers, 1 month and less than $1,000 in compute cost to develop. We hope you will find the learnings useful, as we believe they apply to a wide class of GenAI use cases.”article, stated
The training dataset contained about 3,600 generated examples.
“Then we synthesized CREATE TABLE statements using the above use cases to yield a diverse set of tables and generated sample responses including table descriptions and column comments using another LLM. In total, we generated ~3600 training examples.”article, stated
Fine-tuning on about 3,600 examples took around 15 minutes.
“* The smaller model could fit comfortably and be served using A10 GPUs, which were more readily available. Their abundance would mean higher inference throughput for the task. The total time it took to finetune the model on the ~3600 examples was only around 15 minutes!”article, stated
The final model reduced inference cost by more than 10 times.
“The fine-tuning compute cost for the whole project was less than $1000 (each fine-tuning run cost only a few dollars). And the final result is a more than 10-fold reduction in cost.”article, stated

Alternatives considered (roads not taken)

Off-the-shelf SaaS LLMs

Rejected because: They created quality drift, quota bottlenecks, and costs that did not fit full-catalog production use.

“As we moved towards launching this feature to all our customers, we ran into three challenges with the model: 1. Quality”article, stated

Larger MPT-30B and Llama-2-13B models

Rejected because: The smaller 7B fine-tuned models met the quality bar and larger models did not justify higher serving costs.

“We did not yet observe a measurable benefit of using larger models for this task that would justify the increased serving costs.”article, stated

Customer data for training

Rejected because: Databricks intentionally used public NAICS and internal taxonomy data, not customer data.

“Notably, we didn’t use any customer data for training this powerful feature that all of our customers can benefit from.”article, stated

Patterns

`fine-tuning-adaptation` (model-ops)

Databricks fine-tuned MPT-7B on task-specific synthetic documentation examples to replace general SaaS LLMs.

“In the end we chose MPT-7B, as it has the best combination of quality and inference performance”article, stated

`synthetic-data-generation` (data)

The team generated CREATE TABLE statements and sample documentation from public and internal taxonomy sources.

“we synthesized CREATE TABLE statements using the above use cases to yield a diverse set of tables and generated sample responses”article, stated

`eval-driven-development` (evaluation)

Model selection used double-blind employee evaluation before production rollout and acceptance metrics afterward.

“we set up a simple double-blind evaluation framework in which we asked 4 employees to rate table descriptions generated from the two models”article, stated

`data-quality-loop` (operations)

Production acceptance metrics feed back into model monitoring and iteration.

"Collect real-world usage metrics, so we can monitor how well a monitor does in production

Iterate and roll out new models"

— article, stated

Tradeoffs

Model ownership

Chose bespoke fine-tuned model over off-the-shelf SaaS LLMs.

Rationale: The bespoke model was better, faster, cheaper, and controllable for a narrow documentation task.

“creating a bespoke LLM that is better, faster, and cheaper.”article, stated

Model size

Chose MPT-7B over larger MPT-30B or Llama-2-13B.

Rationale: Fine-tuned 7B models met the quality bar with lower latency and serving cost.

“The latency for the smaller models was significantly better than the larger models while offering comparable quality”article, stated

Training data policy

Chose public and internal taxonomy-derived synthetic data over customer data.

Rationale: This let Databricks train a customer-facing feature without using customer data.

“Notably, we didn’t use any customer data for training this powerful feature that all of our customers can benefit from.”article, stated

Gotchas (what bit them)

SaaS LLM quality can regress without local code changes.

Root cause: The provider may roll out model updates that affect a narrow production task.

Resolution: Build and evaluate a bespoke model controlled by Databricks.

“we also sometimes noticed the quality of the suggestions degrading, without any change to our codebase.”article, stated

General-purpose SaaS LLMs did not fit full-catalog throughput and cost requirements.

Root cause: Large general models require more parameters, quota, and token cost than the narrow task needs.

Resolution: Fine-tune a smaller model that fits A10 GPUs and can scale horizontally.

“The generality means it needs to have an extremely large number of parameters, which limits how fast and how cheap it can return answers.”article, stated

Production evaluation cannot wait until after launch.

Root cause: Acceptance-rate metrics only exist after users can interact with the feature.

Resolution: Use a double-blind pre-launch evaluation on unseen tables.

“After the feature launch, we could measure a model’s quality through production metrics such as the rate of users accepting the suggestions. But before we made it to the launch, we needed a way to evaluate”article, stated

Evaluation

Databricks used production acceptance rates after launch, and before launch used a double-blind human evaluation where 4 employees compared outputs from two models on 62 unseen tables.

Metrics: user acceptance rate · 4 human evaluators · 62 unseen tables · evaluator agreement · quality · throughput · cost

“we asked 4 employees to rate table descriptions generated from the two models we wanted to compare using a set of 62 unseen tables.”article, stated

“Our framework then generated a sheet where each row showed the input and showed both outputs in a randomized order.”article, stated

Outcomes

More than 80 percent of table metadata updates on Databricks are AI-assisted.
The bespoke model was developed by two engineers in one month for less than $1,000 in compute.
The final system delivered more than a 10-fold inference cost reduction.
The model is live on Databricks in AWS and Google Cloud.

Databricks reports >80% AI-assisted table metadata updates, a 3,600-example training set, 15-minute fine-tuning, and >10x inference-cost reduction.

“This model is now live on Databricks in Amazon Web Services and Google Cloud and is being used to power most data annotations on the platform.”article, stated

“And the final result is a more than 10-fold reduction in cost.”article, stated