LLM-Powered GraphQL Mock Data Generation for Developer Productivity

airbnb · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: GraphQL developer productivity and mock data generation

Airbnb needed realistic, type-safe GraphQL mock data that stays synchronized with evolving queries, design context, and client/server parallel development.

Constraints:

Manual GraphQL mocks are tedious to write and maintain.
Client engineers often need to prototype before server implementations are complete.
Mocks drift away from production reality as queries and schemas evolve.
Generated data must validate against the GraphQL schema and feel realistic enough for demos.

Architecture

Airbnb added @generateMock and @respondWithMock directives into its Niobe GraphQL code-generation workflow. Niobe gathers the relevant query/schema subset, documentation, optional design snapshot, and seed image URLs, sends that context to Gemini 2.5 Pro, validates the JSON with GraphQL tooling, feeds validation errors back for self-healing, and maintains generated mocks with query hashes and diffs so unchanged mocks are not needlessly rewritten.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  Generate_directive["@generateMock directive"] --> Niobe_CLI["Niobe CLI"]
  Niobe_CLI --> Context_collector["Context collector"]
  Context_collector --> Design_API["Design snapshot API"]
  Context_collector --> Seed_images["Seed image URL set"]
  Context_collector --> Gemini_model["Gemini 2.5 Pro"]
  Gemini_model --> GraphQL_validator["GraphQL validator"]
  GraphQL_validator --> Gemini_model
  GraphQL_validator --> Hash_manager["Mock hash manager"]
  Hash_manager --> Respond_directive["@respondWithMock directive"]
  Respond_directive --> Niobe_CLI

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`@generateMock directive`	Client GraphQL annotation that requests automatic mock-data generation for an operation, fragment, or field.	`Niobe CLI`, `Context collector`
`Niobe CLI`	Airbnb GraphQL code generator that runs mock generation as part of local code generation.	`@generateMock directive`, `Context collector`, `GraphQL validator`, `@respondWithMock directive`
`Context collector`	Builds the LLM context from the queried schema subset, documentation, design snapshot, and seed image URLs.	`Gemini 2.5 Pro`, `Design snapshot API`, `Seed image URL set`
`Design snapshot API`	Produces an image URL for the relevant design node so the model can match UI expectations.	`Context collector`
`Seed image URL set`	Approved Airbnb-hosted image URLs that prevent hallucinated image references in mocks.	`Context collector`
`Gemini 2.5 Pro`	LLM that generates realistic mock JSON from GraphQL and product context.	`Context collector`, `GraphQL validator`
`GraphQL validator`	graphqlSync validation that catches schema errors and feeds them back to the LLM for repair.	`Gemini 2.5 Pro`, `Mock hash manager`
`Mock hash manager`	Stores client-entity and query hashes to decide whether existing mocks need regeneration.	`GraphQL validator`, `@respondWithMock directive`
`@respondWithMock directive`	Runtime client annotation that returns generated mock data instead of server data for unimplemented fields or queries.	`Niobe CLI`, `Mock hash manager`

Data flow

An engineer adds @generateMock to a GraphQL operation and runs Niobe code generation. Niobe collects the needed schema subset, docs, design image context, and safe image URLs, streams a Gemini 2.5 Pro mock JSON response, validates it with graphqlSync, retries with validation errors when needed, and stores hashes so future codegen updates only changed mocks.

3 supporting quotes

“To generate mock data using @generateMock, engineers simply need to run Niobe code generation after adding the directive — just as they would after making any other GraphQL change.”article, stated

“Next, the CLI aggregates all the context described in the section above — including the URL of the design snapshot — and crafts a prompt to send to the LLM.”article, stated

“Once the mock JSON response has been received from the LLM, Niobe performs a validation step against this data by passing the GraphQL schema, client GraphQL document, and JSON data”article, stated

Why this architecture

Airbnb placed LLM generation inside existing GraphQL infrastructure because realistic mocks need product context, but production safety requires type and schema validation. Keeping generation in Niobe lets engineers stay in their local workflow and gives the system a built-in place to validate, self-heal, and update mocks only when query hashes change.

3 supporting quotes

“By placing the LLM within our existing GraphQL infrastructure, we’re able to enforce a set of guardrails through this validation step and provide strong guarantees”article, stated

“The context that we provide to the LLM is vital to generating data that is realistic enough to use in demos.”article, stated

“At Airbnb, engineers use a command line tool we call Niobe to generate code for their GraphQL queries and fragments.”article, stated

Models used: Gemini 2.5 Pro · LLMs

Scale

Airbnb engineers generated and merged more than 700 mocks across iOS, Android, and Web.
“In just the past few months, Airbnb engineers have generated and merged over 700 mocks across iOS, Android, and Web using @generateMock”article, stated
Gemini 2.5 Pro was selected partly for its 1-million-token context window.
“We chose this model because of its 1-million token context window, plus the fact that in our internal tests this configuration performed significantly faster”article, stated
GraphQL queries can reach hundreds of lines, making hand-crafted mocks costly.
“GraphQL queries can grow to hundreds of lines, and hand-crafting mock response data is extremely tedious.”article, stated

Alternatives considered (roads not taken)

Manual JSON or generated-type mocks

Rejected because: They consume engineer time, are error-prone, and require manual updates as queries change.

“Most engineers manually write mocks as either raw JSON files or by instantiating types generated from the GraphQL schema”article, stated

“they require significant time investment and are prone to subtle mistakes.”article, stated

Random value generation and field-level stubbing

Rejected because: They lack the domain and product context needed to make mocks realistic.

“attempts to improve the process, such as random value generation and field-level stubbing, fall short because they lack essential domain context”article, stated

Always request production server data while developing clients

Rejected because: Client teams often need to build and test UI before the server implements the schema.

“This parallelization is particularly challenging for client engineers, since they can’t actually test the UI they’re building until the server has fully implemented the schema.”article, stated

Patterns

`structured-output-contract` (guardrails)

Generated mock JSON is validated against the GraphQL schema and client document before it becomes usable.

“Niobe performs a validation step against this data by passing the GraphQL schema, client GraphQL document, and JSON data to the graphql NPM package’s graphqlSync function.”article, stated

`context-engineering` (prompting)

Niobe deliberately curates only the needed schema subset, documentation, design snapshot, and safe image URLs into the LLM prompt.

“Importantly, this isn’t the whole schema, because including the full schema would likely overload the context window — Niobe traverses the schema and strips out types and fields”article, stated

“This prevents the LLM from hallucinating image URLs that don’t exist and ensures that the mock data contains valid URLs”article, stated

`eval-driven-development` (evaluation)

Validation errors are fed back to the LLM so invalid mock data can self-heal before being accepted.

“If the validation produces errors (for example, if the LLM hallucinated an invalid enum value or failed to populate a required field), Niobe aggregates these errors”article, stated

`tool-use-orchestration` (orchestration)

The LLM-powered generator is embedded inside Niobe codegen and runtime GraphQL directives rather than used as a separate chat tool.

“When an engineer uses the Niobe CLI to generate code for their GraphQL files, Niobe automatically performs mock generation as the final step”article, stated

Tradeoffs

Generate mocks inside GraphQL codegen

Chose Niobe-integrated @generateMock over external chat/manual mock generation.

Rationale: The integrated path preserves local developer flow and lets existing GraphQL validation enforce correctness.

“engineers simply need to run Niobe code generation after adding the directive — just as they would after making any other GraphQL change.”article, stated

“something that wouldn’t be possible by using a tool outside our GraphQL infrastructure like ChatGPT.”article, stated

Use curated production-like assets instead of letting the model invent everything

Chose schema subset, design snapshots, and Airbnb-hosted image URLs over unconstrained mock generation from a prompt alone.

Rationale: The curated context improves realism and prevents invalid image URLs.

“A list of Airbnb-hosted image URLs that the LLM can choose from if needed, along with short textual descriptions of each.”article, stated

Gotchas (what bit them)

Generated mocks can be invalid even when they look plausible.

Root cause: LLMs can hallucinate invalid enum values or omit required fields.

Resolution: Validate with GraphQL infrastructure and feed validation errors back to the LLM for repair.

“if the LLM hallucinated an invalid enum value or failed to populate a required field), Niobe aggregates these errors and feeds them back into the LLM”article, stated

Full schemas can overload context windows.

Root cause: GraphQL schemas contain many unrelated types and fields not needed for a specific mock.

Resolution: Traverse the schema and strip out types, fields, and whitespace unnecessary to resolve the query.

“including the full schema would likely overload the context window — Niobe traverses the schema and strips out types and fields that are not needed”article, stated

Regeneration can overwrite useful manual tweaks or break tests.

Root cause: Naively regenerating every mock ignores whether queried fields actually changed.

Resolution: Track hashes and pass only diffs plus existing mocks when changes require updates.

“It’s important that Niobe doesn’t unnecessarily modify existing mock data for fields that are unchanged and still valid”article, stated

Evaluation

Airbnb evaluated realism and speed in internal tests, validates every generated mock against GraphQL, and reports adoption through merged mocks across platforms.

Metrics: over 700 mocks generated and merged · GraphQL validation of generated JSON · Gemini 2.5 Pro faster than comparable models in internal tests

“We chose this model because of its 1-million token context window, plus the fact that in our internal tests this configuration performed significantly faster”article, stated

“In just the past few months, Airbnb engineers have generated and merged over 700 mocks across iOS, Android, and Web using @generateMock”article, stated

Outcomes

Engineers can generate valid, realistic GraphQL mocks from local codegen.
Client teams can prototype against generated mocks before server logic exists.
More than 700 mocks were generated and merged across iOS, Android, and Web.

The system is deployed across Airbnb client platforms and is planned for backend-service support.

“The directive-driven approach of @generateMock and @respondWithMock allows engineers to build clients before the server implementation is complete”article, stated

“These tools have fundamentally changed how engineers mock GraphQL data for tests and prototypes at Airbnb”article, stated