Automating Code Sample Updates with LLMs for Technical Documentation

wix · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Technical documentation maintenance

Wix needed to update thousands of Velo API reference code samples after a syntax change without stopping the technical writing team's normal work.

Constraints:

• the API reference contained more than 2,000 code samples
• only a subset used the affected web-method export/import pattern
• code samples varied in structure and style after years of evolution
• source files lived across multiple repositories
• converted samples had to compile and preserve unaffected code

Architecture

Wix built a docs-as-code migration pipeline. A mapping step found source files for API reference samples. GPT-4 classified which samples needed rewriting, and GPT-3.5 Turbo converted affected samples to the new webMethod syntax. A TypeScript compiler-based validator checked whether converted samples used Velo APIs correctly. Writers reviewed validation errors, fixed edge cases, and a final script applied conversions back to source repositories and opened PRs.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  sample-mapping-script[Sample mapping script] --> classification-script[Classification script]
  classification-script --> conversion-script[Conversion script]
  conversion-script --> typescript-validator[TypeScript validator]
  typescript-validator --> writer-error-review[Writer error review]
  writer-error-review --> replacement-script[Replacement script]

Components

Data flow

The workflow collects code samples and source mappings, classifies each sample with GPT-4, converts selected samples with GPT-3.5 Turbo, stores rewritten files beside originals for comparison, validates converted samples with TypeScript compilation and Velo type definitions, routes validation errors to writers, and applies accepted conversions back to source repositories.

Why this architecture

Wix split the job into classification, conversion, validation, and replacement because the affected samples could not be found with a simple script, and converted code could not be published without compilation checks. Using GPT-4 for classification and GPT-3.5 Turbo for conversion reflects task-specific model choice: classification needed robust semantic filtering, while conversion worked best with the cheaper/faster model they tested.

Models used: GPT-4 · GPT-3.5 Turbo

Scale

• The Velo API reference contained more than 2,000 code samples.
  “Wix’s Velo API reference contains over 2,000 code samples. About a year ago, we introduced a syntax change to Velo that suddenly rendered a huge number of these samples outdated.”article, stated
• The source files were spread across Wix's technical-writing repo and many API source repositories.
  “Other files are dispersed among Wix’s many repos that also house the source code for the APIs. There was no straightforward way (that we could find) to identify the source file for any given code sample.”article, stated
• The final human review was limited to six writers working one morning.
  “By automating the process, we were able to limit team involvement to 6 writers who worked on the project for one morning.”article, stated

Alternatives considered (roads not taken)

Manually update every affected sample

Rejected because: Manual work would have consumed large writing-team effort and halted normal velocity.

Script pattern detection without an LLM

Rejected because: Wix had no programmatic way to identify affected samples because formats and repositories varied.

Publish converted samples without compiling

Rejected because: LLM output could be wrong, and bad code could not ship in reference docs.

Patterns

model-routing (architecture)

Wix used GPT-4 for classification and GPT-3.5 Turbo for conversion after testing task fit.

eval-driven-development (evaluation)

TypeScript compilation validated converted samples before publication.

batch-offline-processing (operations)

Scripts classified, converted, validated, and replaced thousands of documentation samples in batch.

human-in-the-loop-gate (workflow)

Writers reviewed validation errors before converted samples were merged back into source repositories.

prompt-versioning (prompting)

Wix used explicit system prompts and examples for both classification and conversion.

Tradeoffs

Use LLM classification despite probabilistic errors

Chose GPT-4 classifier tested on a sample set over manual discovery or brittle scripts.

Rationale: The team judged LLM classification acceptable because there was no easy deterministic identifier and the prompt failed only a small number of tests.

Validate converted code with TypeScript

Chose compiler-backed validation over trusting the conversion prompt.

Rationale: Compilation provided a deterministic check that converted samples remained valid JavaScript using Velo APIs correctly.

Use GPT-3.5 Turbo for conversion

Chose GPT-3.5 Turbo over other available models.

Rationale: Wix found it produced the best conversion results in their experiments.

Gotchas (what bit them)

The affected samples could not be identified by simple syntax search.

Root cause: Only some samples included web-method exports, formats varied, and source files were spread across repos.

Resolution: Wix used an LLM classifier to flag samples requiring conversion.

LLM conversions could fail for non-semantic reasons.

Root cause: The model sometimes hit return-token limits, and type definitions could be out of sync.

Resolution: The pipeline logged validation errors for manual review.

Converted documentation code must preserve unaffected code.

Root cause: The syntax change only required specific import/export transformations.

Resolution: The prompt explicitly instructed the model to change only the required elements.

Evaluation

Wix evaluated classification prompt performance on a testing data set and validated converted code with TypeScript compilation plus manual review of validation errors.

Metrics: classification test failures · compile validation errors · manual validation workload · writer time saved

Outcomes

• The project updated code samples without consuming the full writing team.
• Human involvement was limited to six writers for one morning.
• The LLM did not produce observed conversion-semantics failures in validation; errors were cutoffs or stale type definitions.
• The pipeline returned PRs for publishing updated docs.

The migration covered a reference with more than 2,000 code samples and reduced manual effort to a small writer review pass.