Cloud-Native Synthetic Data Generator for Data Pipeline Testing

godaddy · Tech · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise data-platform testing and synthetic data generation

GoDaddy needed realistic test data for data pipelines without copying production data into lower environments or forcing engineers to handcraft datasets for many schemas.

Constraints:

• Production data could not be copied into development, testing, or staging because of privacy and security concerns.
• Manual test-data creation did not scale to tens or hundreds of maintained schemas.
• Pure LLM row generation was too expensive and slow for millions or billions of records.
• Generated data still needed schema compliance, realistic distributions, referential integrity, cross-account delivery, and operational observability.

Architecture

GoDaddy built a cloud-native synthetic data generator that uses GoCode, its internal LLM API, to interpret submitted schemas and create Databricks Labs Datagen templates, then runs Spark jobs on EMR Serverless to generate schema-conformant Parquet data at scale. The Data Lake API records schema and target metadata in DynamoDB, DLMS orchestrates validation, idempotency, retries, and EMR submission, and generated outputs are delivered directly into producer-owned S3 buckets with status tracking.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  producer-or-consumer-client["producer or consumer client"] --> data-lake-api["Data Lake API"]
  data-lake-api --> dynamodb-status-store["DynamoDB status store"]
  data-lake-api --> dlms-orchestrator["DLMS orchestrator"]
  dlms-orchestrator --> lambda-job-submitter["Lambda job submitter"]
  lambda-job-submitter --> gocode-template-engine["GoCode template engine"]
  gocode-template-engine --> template-validation-layer["template validation layer"]
  lambda-job-submitter --> emr-serverless-spark["EMR Serverless Spark"]
  template-validation-layer --> emr-serverless-spark
  gocode-template-engine --> emr-serverless-spark
  emr-serverless-spark --> producer-s3-bucket["producer S3 bucket"]
  emr-serverless-spark --> quality-validation-checks["quality validation checks"]
  quality-validation-checks --> dynamodb-status-store
  producer-s3-bucket --> producer-or-consumer-client

Components

Data flow

A producer or consumer submits a schema and generation parameters through the Data Lake API. The API persists metadata and desired state in DynamoDB, DLMS starts the workflow, a Lambda builds the EMR job specification, GoCode renders a strongly typed Datagen/Spark template from the schema, EMR executes partitioned Spark generation, and the resulting compressed Parquet files are written to the producer's S3 prefix while status updates flow back through DynamoDB and DLMS.

Why this architecture

The architecture deliberately uses the LLM for schema understanding and template creation, not for per-row generation, because GoDaddy needed LLM-level semantic inference with traditional distributed-generation economics, security, and throughput.

Models used: GoCode internal LLM API · Databricks Labs Datagen · Spark on EMR Serverless

Scale

• The system targets GoDaddy's many data streams and pipelines.
  “At GoDaddy, we are a data-driven company that relentlessly pursues valuable insights through our various data streams and pipelines.”article, stated
• Manual schema support had to cover tens to hundreds of schemas.
  “Especially when tens to hundreds of schemas are expected to be created and maintained; this expectation doesn’t scale.”article, stated
• The platform is designed for billions of generated rows.
  “EMR Serverless: Provides the distributed compute needed to generate billions of rows efficiently.”article, stated
• Pure LLM generation was estimated to cost thousands of dollars and take days for one million customer records.
  “generating one million customer records would cost thousands of dollars and take days to complete.”article, stated
• The hybrid template approach claims a 99.9% cost reduction versus pure LLM generation.
  “a 99.9% cost reduction compared to pure LLM generation.”article, stated
• The Lambda prototype hit the 15-minute timeout at million-row scale.
  “This worked for small datasets but hit the 15-minute timeout limit when generating millions of rows.”article, stated
• EMR Serverless batching reduced costs by 80% versus always-on clusters.
  “We adopted EMR Serverless with intelligent job batching, reducing costs by 80% compared to always-on clusters.”article, stated

Alternatives considered (roads not taken)

Manual test data creation

Rejected because: Engineers spent days writing JSON and SQL, and the approach could not scale to hundreds of schemas and millions of records.

Production data sampling and anonymization

Rejected because: It introduced security and compliance risks and could not generate future-state scenarios.

Off-the-shelf generators alone

Rejected because: Generic tools handled simple schemas but struggled with GoDaddy-specific relationships, constraints, and business logic.

Pure LLM generation of entire datasets

Rejected because: Quality was promising, but the cost and latency of generating millions of rows were impractical.

Lambda-based data generation

Rejected because: Lambda worked for small datasets but timed out on million-row jobs, pushing the team to EMR Serverless.

Always-on EMR clusters

Rejected because: The team judged them too expensive for per-request generation and chose EMR Serverless with batching.

Patterns

synthetic-data-generation (data-generation)

GoDaddy generates production-like test datasets from schemas so pipeline teams can test without production data exposure.

structured-output-contract (reliability)

The LLM produces structured Datagen templates that are checked against schema constraints before execution.

batch-offline-processing (execution)

Large generation requests run as distributed EMR/Spark jobs and small requests are batched into shared jobs.

cost-aware-routing (optimization)

The system routes intelligence to one-time template generation and routes row generation to cheaper distributed compute.

data-quality-loop (reliability)

Generated datasets are validated for schema, constraints, statistics, and referential integrity before being trusted for pipeline tests.

privacy-preserving-processing (security)

Synthetic data removes the need to copy sensitive production records into lower environments.

observability-tracing (operations)

Every generation job records metadata and error information in DynamoDB for debugging and monitoring.

Tradeoffs

Use LLMs for template generation rather than row generation.

Chose LLM-generated Datagen templates executed by Spark over LLMs generating entire datasets directly.

Rationale: Template-time LLM use gives semantic understanding while avoiding per-row cost and latency.

Use EMR Serverless for generation jobs.

Chose EMR Serverless with Spark over Lambda functions for all generation.

Rationale: EMR Serverless supports long-running, horizontally scalable jobs after Lambda timed out on large datasets.

Deliver output into producer-owned buckets.

Chose Cross-account write via bucket policy or assumed role over Central platform-owned storage or copied handoff.

Rationale: Producer-owned delivery aligns with ownership and zero-trust security practices but requires flexible permission handling.

Expose synthetic-data generation as an API-first self-service platform.

Chose Simple API calls hiding the underlying LLM, Spark, EMR, S3, and validation complexity over Specialist-operated generation workflows.

Rationale: Self-service lowers adoption friction for teams that need data without understanding synthetic-data internals.

Gotchas (what bit them)

LLM-generated templates can be syntactically or logically invalid.

Root cause: The LLM may hallucinate code or constraints when translating schemas into generation templates.

Resolution: Validate templates against schema constraints and known-good patterns, then regenerate invalid outputs with more specific prompts.

Serverless functions are too small for million-row generation jobs.

Root cause: The Lambda prototype hit the 15-minute execution limit when data volume grew.

Resolution: Move generation to EMR Serverless and rewrite logic in Spark for long-running horizontal execution.

Cross-account delivery can block synthetic-data adoption.

Root cause: Generated data must be written into producer-owned S3 buckets across AWS accounts, creating permission complexity.

Resolution: Support both bucket policies and assumed roles so producers can choose the model that fits their security posture.

Synthetic data can still miss production messiness.

Root cause: Generated data may not capture subtle real-world correlations, typos, incomplete records, or unexpected relationships.

Resolution: Treat synthetic data as one testing tool and periodically compare synthetic distributions with production patterns.

Always-on clusters make occasional generation requests too expensive.

Root cause: Running dedicated EMR capacity for every request wastes resources when demand is bursty.

Resolution: Use EMR Serverless and batch small requests into shared jobs.

Evaluation

The article evaluates the system through operational design checks, generated-data validation stages, and reported impact metrics from early use rather than a formal model benchmark.

Metrics: schema compliance · constraint validation · statistical validation · referential integrity · 90% reduction in time spent creating test data · 100% elimination of production data in test environments · 5x faster pipeline development cycles · 99.9% LLM-cost reduction versus pure LLM generation · 80% infrastructure-cost reduction versus always-on EMR clusters

Outcomes

• GoDaddy built a cloud-native synthetic data generator that converts schemas into production-like datasets through an API call.
• The hybrid architecture reduced LLM costs by using LLMs once per schema rather than for every row.
• EMR Serverless and Spark enabled large-scale generation beyond Lambda's runtime limits.
• The system reports reduced data-creation time, eliminated production-data use in tests, and faster pipeline development.

The article reports billions-of-rows design intent, hundreds-of-schemas pressure, 99.9% LLM-cost reduction, 80% lower EMR cost versus always-on clusters, 90% lower test-data creation time, 100% removal of production data from test environments, and 5x faster pipeline development cycles.