Large-Scale Enterprise Data Platform Migration Using AI and Generative AI Automation

commbank · Finance · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Regulated enterprise data-platform migration

Commonwealth Bank of Australia needed to move decades of on-premise data platforms into an AWS data mesh so federated data scientists and engineers could work where the data lived and build AI systems at scale. The migration had to transform tens of thousands of legacy pipelines, preserve exact financial-data correctness, satisfy governance and risk controls, and complete quickly enough that the bank's AI strategy was not blocked by legacy infrastructure.

Constraints:

• CBA serves 17.5 million customers and processes about half of Australia's transactions
• legacy on-premise platforms held years of data across hundreds of source systems
• the old platforms lacked interoperability and could not scale for AI
• 61,000 on-premise pipelines and about 10 petabytes of data had to move to AWS
• every row, column, and number needed exact reconciliation to the on-premise platform
• governance and risk controls were non-negotiable for a regulated bank
• the program had an ambitious nine-month timeline and required coordinated CBA, AWS, and HCL execution

Architecture

CBA built CommBank.data, an AWS-based data mesh with a federated producer-consumer operating model, self-service data marketplace, and embedded governance. The migration program used steel-thread workshops with AWS and HCL to prove and productionalize migration paths for complex pipelines and AI use cases. AI and generative AI accelerators transformed legacy pipeline code, checked for errors, and supported output reconciliation. Parallel testing and automation validated migrated outputs against on-premise tables, while HCL scaled certified data-engineering teams, connect pods, migration squads, automation squads, and embedded AI engineers under a unified delivery cadence.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  onprem["On-premise data platforms"] --> steel["Steel-thread validation"]
  onprem --> accelerators["AI migration accelerators"]
  steel --> accelerators
  accelerators --> recon["Parallel reconciliation harness"]
  recon --> mesh["AWS mesh ecosystem"]
  accelerators --> mesh
  mesh --> cbdata["CommBank.data"]
  cbdata --> teams["Producer-consumer data teams"]
  cbdata --> marketplace["Unified data marketplace"]
  cbdata --> governance["Governance safeguards"]
  teams --> marketplace
  engineers["Embedded AI engineers"] --> accelerators
  delivery["Migration delivery teams"] --> engineers
  delivery --> mesh

Components

Data flow

Legacy on-premise pipeline logic and data moved through steel-thread validation, AI-assisted code transformation, error checking, and parallel test/reconciliation loops. The migrated pipelines landed in the AWS mesh ecosystem, where business units own data products under a producer-consumer model. Users discover and request data through a unified marketplace, while governance and risk controls are embedded across the data and AI lifecycle. Engineering teams used metrics, certification, full-cycle engineering, and AI-driven automation to keep migration quality improving sprint by sprint.

Why this architecture

CBA chose a governed AWS data mesh because centralized data teams and on-premise platforms could not keep pace with AI-scale business demand. The steel-thread approach reduced risk by proving and productionalizing the hardest migration paths early. AI accelerators and parallel testing were necessary because the scope and timeline made manual transformation of 61,000 pipelines impractical, while the bank's regulatory context required exact reconciliation rather than best-effort migration.

Models used: AI and generative AI code-transformation accelerators · agent-led delivery cycle (future direction) · AWS cloud · CommBank.data

Scale

• CBA serves 17.5 million customers.
  “CBA services 17.5 million customers”transcript, stated
• The bank represents one in three Australians and one in four businesses.
  “one in three Australians and one in four businesses bank with us.”transcript, stated
• CBA processes about 50% of Australia's transactions.
  “Overall, 50% of Australia's transactions go through CBA.”transcript, stated
• CommBank.data empowers 40 lines of business.
  “Today, it empowers 40 lines of business”transcript, stated
• The migration moved 61,000 on-premise pipelines.
  “We migrated 61,000 on-premise pipelines”transcript, stated
• The migrated data estate was equivalent to 10 petabytes.
  “That's equivalent to 10 petabytes of data.”transcript, stated
• The migration completed in nine months.
  “The migration took nine months”transcript, stated
• Every data pipeline was tested three times, totaling 229,000 tests.
  “with a hundred percent of data pipelines tested three times, that's 229,000 tests.”transcript, stated
• HCL's delivery team peaked at more than 250 AWS-certified data engineers.
  “peaking at over 250 AWS-certified data engineers.”transcript, stated

Alternatives considered (roads not taken)

Keep data engineering on centralized on-premise platforms

Rejected because: The centralized team and on-premise systems could not keep pace with AI demand or scale for AI workloads.

Use a traditional proof of concept or MVP

Rejected because: CBA used steel threads because they prove the technology and productionalize the outcome.

Rely on manual code migration alone

Rejected because: The migration used AI and generative AI to transform code, check errors, and test outputs at 61,000-pipeline scale.

Centralize all data ownership after migration

Rejected because: CBA decentralized ownership into a producer-consumer model where each business unit manages its data as a product.

Patterns

batch-offline-processing (migration)

The migration transformed and reconciled a large population of legacy data pipelines as a controlled offline modernization program.

eval-driven-development (quality)

CBA required migrated outputs to reconcile exactly, tested every pipeline three times, and used quality metrics sprint by sprint.

tool-use-orchestration (automation)

AI and generative AI accelerators were embedded into the migration workflow to transform code, check errors, and test output.

safety-policy-enforcement (governance)

Governance and risk controls were embedded through the data and AI lifecycle as safety by design for a regulated bank.

data-quality-loop (data-platform)

The migration treated exact table reconciliation and sprint-level quality metrics as feedback loops for improving migration quality.

Tradeoffs

Data-platform operating model

Chose Federated data mesh with embedded business data teams over A centralized data team model.

Rationale: CBA says centralized data could not keep pace with AI demand, so data engineers and scientists moved closer to lines of business and customers.

Migration proof strategy

Chose Steel threads over Traditional POCs or MVPs.

Rationale: Steel threads were used to prove the technology while also productionalizing the migration outcome.

Migration acceleration approach

Chose AI and generative AI automation plus parallel testing over Manual migration and sequential validation alone.

Rationale: The scale of 61,000 pipelines, 10 petabytes, and a nine-month timeline required automated transformation and parallel quality checks.

Governance placement

Chose Embed safeguards throughout the data and AI lifecycle over Treat compliance as a separate post-migration review.

Rationale: The regulated-bank context made governance non-negotiable and required safety by design.

Gotchas (what bit them)

AI-scale data work follows the data location.

Root cause: Data scientists and engineers need to work where enterprise data lives, so leaving rich data on-premise blocks cloud AI adoption.

Resolution: Move the data estate and workforce onto the AWS mesh ecosystem rather than only migrating selected applications.

A large bank cannot accept approximate data migration correctness.

Root cause: Financial data accuracy, regulatory trust, and operational integrity depend on exact table-level consistency.

Resolution: Reconcile migrated outputs exactly to on-premise tables and account for every row, column, and number.

Traditional POCs may prove feasibility without reducing production migration risk.

Root cause: The hardest pipelines and AI use cases need productionization evidence, not only a demo.

Resolution: Use steel threads to test complex pipelines and productionalize the outcome early.

Large migration programs can fail through partner misalignment.

Root cause: Ambitious deadlines and big programs create coordination risk across bank, cloud, and systems-integrator teams.

Resolution: Operate as one unified team with shared purpose, cadence, ownership, and close CBA-AWS-HCL collaboration.

AI migration accelerators still require organizational capacity building.

Root cause: Automation works inside an engineering delivery model, not as a replacement for trained data engineers.

Resolution: Upskill early, certify engineers, shift toward full-cycle engineering, and embed AI engineers in teams.

Evaluation

CBA evaluated the migration through steel-thread tests on complex pipelines and AI use cases, exact output reconciliation to the on-premise platform, three tests for every pipeline, sprint-level quality metrics, and delivery-progress tracking.

Metrics: 61,000 migrated pipelines · 10 petabytes of data · nine-month migration duration · 100% of data pipelines tested three times · 229,000 total tests · 100% table reconciliation to the on-premise platform · every row, column, and number accounted for · over 250 AWS-certified data engineers · 40 lines of business empowered

Outcomes

• CBA migrated 61,000 on-premise pipelines, equivalent to about 10 petabytes of data, into an AWS mesh ecosystem.
• The migration completed in nine months with 100% of pipelines tested three times and 229,000 total tests.
• CommBank.data now empowers 40 lines of business to produce and consume data within a governed framework.
• The program moved CBA's data engineering and AI workforce to AWS cloud.
• HCL reported scaling the engineering team to more than 250 AWS-certified data engineers.
• CBA and HCL plan to use AI accelerators for future transitions and legacy migrations.

The transcript describes a 17.5-million-customer bank handling 50% of Australia's transactions, 40 lines of business on CommBank.data, 61,000 pipelines, 10 petabytes, nine months, 229,000 tests, 100% reconciliation, and a delivery team peaking above 250 AWS-certified data engineers.