Building and Evaluating Maya: An AI-Powered Data Pipeline Generation System

maia · Tech · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: AI-powered data pipeline generation

Matillion needed Maya to help data engineers create data pipelines from natural-language requests, then mature the product from impressive demos into an enterprise-ready system with reliable evaluation and observability.

Constraints:

Data pipeline generation must produce valid DPL and pass component validation.
The product moved from vibes-based judgment to enterprise confidence requirements.
Evaluation traces can contain PII from sample pipeline data.
Traditional software engineers needed LLMOps-style evaluation and observability workflows.

Architecture

Maya runs on Matillion's Data Productivity Cloud as a team of digital data engineers. It interprets user prompts through a semantic-layer knowledge graph, generates human-readable DPL for data pipelines, uses the same validation capabilities a human data engineer would use, and is evaluated through human review, LLM-as-judge experiments, automated harnesses, and Langfuse observability.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  user-prompt[User Prompt] --> maya[Maya]
  maya --> semantic-layer[Semantic Layer]
  semantic-layer --> dpl-generator[DPL Generator]
  maya --> dpl-generator[DPL Generator]
  dpl-generator --> component-validation[Component Validation]
  component-validation --> evaluation-harness[Evaluation Harness]
  maya --> evaluation-harness[Evaluation Harness]
  evaluation-harness --> langfuse[Langfuse]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`User Prompt`	Natural-language request for a connector, transformation, or pipeline.	`Maya`
`Maya`	Digital data engineer agent that plans and generates pipelines.	`Semantic Layer`, `DPL Generator`, `Component Validation`, `Evaluation Harness`
`Semantic Layer`	Knowledge graph-like layer that helps Maya understand data and pipeline context.	`Maya`, `DPL Generator`
`DPL Generator`	Produces the file format that underpins Matillion data pipelines.	`Maya`, `Component Validation`
`Component Validation`	Validation capability also available to human data engineers.	`DPL Generator`, `Evaluation Harness`
`Evaluation Harness`	Manual and automated tests for Maya outputs.	`Maya`, `Langfuse`
`Langfuse`	Central LLM observability platform for automated outputs and traces.	`Evaluation Harness`

Data flow

A user describes a desired data pipeline, Maya uses the semantic layer to understand available data and translate intent into DPL, validates generated components through Matillion tools, and sends conversations plus automated outputs into evaluation and Langfuse dashboards.

“say user enters a prompt into into a prompt box of what they're trying to do with their data pipeline or what they're trying to do to build their connector or what they're trying to do to to do whatever they want to do in their data pipeline. It produces DPL. DPL”transcript, stated

“what Maya does is using as it says there at the bottom the the semantic layer which is kind of a a knowledge graph. It ultimately produces DPL which is in the top right corner there that kind of YAML based file which is kind of Matilian's file format that ultimately underpins a data pipeline.”transcript, stated

Why this architecture

Maya needed to behave like a data engineer inside Matillion rather than a generic chatbot, so it was built on the Data Productivity Cloud, semantic layer, DPL, and validation tools. The evaluation stack evolved because vibes were not enough for enterprise launch and model upgrades.

“more productive. So Maya runs on top of the data productivity cloud which is ultimately the kind of data management platform that Matilian has built over the years and that that pays for this amazing office and things like that that we have here today and say Maya started”transcript, stated

“based on my vibes I think we're good to go which was was a good line at the time but yeah obviously is not going to work that well when people are paying us say hundreds of thousands of dollars and we need to be able to answer all the”transcript, stated

Models used: Claude Sonnet 3.5 · Claude Sonnet 4.5 · LLM judges

Scale

Maya launched at Snowflake Summit in June 2025.
“this slide. So, but it did launch in June 2025 say at Snowflake Summit. It was without doubt that the best product launch I've ever been involved in. We we're a products company at Matilian.”transcript, stated
Four software engineers built Maya from the ground up over the referenced period.
“literally four software engineers who built kind of Maya from the ground up across this time supported by the likes of Julian and say kind of now a growing army of people as the company kind of real shifts towards it. So it was very much built from a homegrown perspective.”transcript, stated
The team could evaluate a Sonnet upgrade within about 24 hours according to the summary.
“automated testing harnesses integrated with Langfuse for observability. This evolution enabled them to confidently upgrade models (like moving to Claude Sonnet 3.5 within 24 hours) and successfully launch Maya to enterprise customers in June 2024, while navigating challenges around PII”summary, stated

Alternatives considered (roads not taken)

Keep Maya as a generic co-pilot interface.

Rejected because: The product evolved into a digital data engineer integrated with pipeline generation and validation.

“started as a as a co-pilot kind of interface into the DPC and over the last kind of 12 months or so it's absolutely morphed into kind of the core interface into the DPC and kind of the the way that we believe in Matilian all kind of”transcript, stated

Use vibes-based release judgment.

Rejected because: Enterprise customers required measured confidence rather than subjective demo impressions.

“that was fantastic, but we needed to go from those Maya moments to ultimately big enterprise customers giving us hundreds of thousands of pounds to run this in production. And it was at that point where after well a little bit before that because we kind of hoped it”transcript, stated

Rely only on manual human scoring.

Rejected because: The team moved toward automated harnesses and Langfuse observability after initial human-in-the-loop checks.

“ultimately to begin with we very much had a manual kind of human in the loop to be able to both rate the answer to be able to provide their own counterpoints as to what that answer was and to be able to ultimately understand how much confidence did we have in an LLM's”transcript, stated

Patterns

`tool-use-orchestration` (agent tools)

Maya uses the same data-pipeline capabilities and validation tools as a human data engineer.

“Maya has full use of kind of the the same things that a a human being would have a human data engineer”transcript, stated

`structured-output-contract` (pipeline generation)

The agent generates DPL, the human-readable file format that underpins Matillion pipelines.

“file format that ultimately underpins a data pipeline. So as you can see there it's very human readable”transcript, stated

`eval-driven-development` (evaluation)

Maya evolved from certification tests to human review, LLM judges, automated harnesses, and Langfuse dashboards.

“we are maturing. We're getting more confidence in what Maya does. We're at a very very early stage of the journey”transcript, stated

`observability-tracing` (operations)

Automated evaluation outputs flow into Langfuse for central LLM observability.

“All of that output is now going into Langfuse as kind of a central LLM observability platform.”transcript, stated

`privacy-preserving-processing` (data governance)

The team had to address PII exposure in sample pipeline data stored in observability traces.

“sample data in the pipelines that we run there's lots of PII in there. There's lots of data that we don't want”transcript, stated

Tradeoffs

Product shape

Chose Digital data engineer integrated into DPC over Generic chatbot co-pilot.

Rationale: A domain-integrated agent can create actual data-pipeline artifacts and use Matillion validation.

“Maya is a digital data engineer and more specifically Maya is a team of digital data engineers”transcript, stated

Release confidence

Chose Data-backed evaluation and observability over Vibes-based product judgment.

Rationale: Enterprise adoption required repeatable confidence in generated pipelines.

“we needed to go from those Maya moments to ultimately big enterprise customers giving us”transcript, stated

Evaluation automation

Chose LLM judges plus human validation over Pure manual review.

Rationale: Manual review established trust, while automated harnesses made repeated model and prompt evaluation practical.

“manual kind of human in the loop to be able to both rate the answer”transcript, stated

Gotchas (what bit them)

Demo delight did not equal enterprise readiness.

Root cause: Early Maya moments were judged by subjective vibes rather than repeatable evaluation.

Resolution: Build manual and automated evaluation harnesses before relying on the product for enterprise launch.

“based on my vibes I think we're good to go which was was a good line at the time”transcript, stated

Observability traces risked storing sensitive data.

Root cause: Pipeline sample data flowing into Langfuse could contain PII.

Resolution: Account for PII requirements when instrumenting evaluation and trace storage.

“Langfuse was now storing the sample data. Now sample data in the pipelines that we run there's lots of PII in there.”transcript, stated

LLM-as-judge needed calibration to the target model.

Root cause: The team had to assess confidence in an LLM's ability to judge Maya outputs.

Resolution: Use human-in-the-loop ratings to validate judge behavior before automation.

“confidence did we have in an LLM's ability to judge the output of what Maya did based on a set of input”transcript, stated

Evaluation

Maya's evaluation matured from certification-style tests and vibes to human-in-the-loop ratings, LLM-as-judge checks, API-based test harnesses, and Langfuse-observed automated outputs.

Metrics: Component validation · Human ratings · LLM judge agreement · Langfuse output variation · Model-upgrade readiness

“Maya literally talking to itself in terms of the conversation it was having”transcript, stated

“testing an LLM output as part of what we're doing with Maya.”transcript, stated

Outcomes

Maya launched at Snowflake Summit in June 2025.
The team moved from subjective Maya moments to automated evaluation and Langfuse observability.
The system provided enough confidence to support rapid model upgrades and enterprise customers.

The source reports a June 2025 launch, four engineers building Maya, and a shift toward automated evaluation outputs in Langfuse.

“it did launch in June 2025 say at Snowflake Summit.”transcript, stated