Hybrid Cloud Architecture for AI/ML with Regulatory Compliance in Banking

bank_centercredit_(bcc) · Finance · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: regulated banking AI/ML infrastructure, speech recognition, and internal HR support

Bank CenterCredit needed to use cloud AI/ML services and GPU capacity while meeting banking requirements that data be encrypted with locally held keys and customer data be anonymized before processing outside the bank's security perimeter.

Constraints:

• the bank serves more than 3 million clients through 20 branches and more than 150 offices
• data transmission and storage must be encrypted
• encryption keys must be stored inside the bank
• customer data must be anonymized during collection and processing
• existing local infrastructure limited scalability for AI/ML and GPU workloads
• call-center recordings use 8 kHz audio while many ASR models are trained on 16 kHz data
• Kazakh/Russian mixed language had no suitable market ASR model
• HR chatbot data must remain compliant with internal and external regulations

Architecture

BCC built a hybrid multi-cloud foundation using AWS Outposts in its own data centers, managed Kubernetes/EKS, S3 on Outposts, centralized deployment/logging/load-balancing tools, and AWS KMS External Key Store so local keys can protect cloud processing. On top of that platform, the bank built two AI use cases. The ASR pipeline keeps raw call-center data on premise, splits audio into channels and voice chunks, generates text with an existing speech-to-text model, removes PII/sensitive chunks with NER, converts cleaned audio to Mel spectrograms, synchronizes encrypted spectrograms to S3 in the AWS region, fine-tunes in SageMaker with cloud GPUs, and returns artifacts for on-premise inference. The HR bot uses hybrid RAG: local embeddings and a Postgres vector database on Outposts retrieve HR policy context, then prompt plus context is sent through Bedrock to Claude Sonnet and returned to the on-premise chatbot.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  On_premise_bank_data_centers["On-premise bank data centers"] --> AWS_Outposts_racks["AWS Outposts racks"]
  AWS_Outposts_racks --> EKS_on_Outposts["EKS on Outposts"]
  AWS_Outposts_racks --> S3_on_Outposts["S3 on Outposts"]
  AWS_Outposts_racks --> XKS_proxy["XKS proxy"]
  XKS_proxy --> AWS_KMS["AWS KMS"]
  Hadoop_call_recording_cluster["Hadoop call recording cluster"] --> Audio_preprocessing_pipeline["Audio preprocessing pipeline"]
  Audio_preprocessing_pipeline --> NER_anonymization_step["NER anonymization step"]
  NER_anonymization_step --> Mel_spectrogram_converter["Mel spectrogram converter"]
  Mel_spectrogram_converter --> S3_on_Outposts
  S3_on_Outposts --> S3_in_AWS_region["S3 in AWS region"]
  S3_in_AWS_region --> SageMaker_fine_tuning_job["SageMaker fine-tuning job"]
  SageMaker_fine_tuning_job --> ASR_model_artifacts["ASR model artifacts"]
  ASR_model_artifacts --> Local_GPU_inference_pool["Local GPU inference pool"]
  Local_GPU_inference_pool --> Post_call_analytics_pipeline["Post-call analytics pipeline"]
  HR_knowledge_base["HR knowledge base"] --> Local_embedding_model["Local embedding model"]
  HR_chatbot_UI["HR chatbot UI"] --> Local_embedding_model
  Local_embedding_model --> Postgres_vector_database["Postgres vector database"]
  Postgres_vector_database --> Amazon_Bedrock_private_endpoint["Amazon Bedrock private endpoint"]
  Amazon_Bedrock_private_endpoint --> Claude_Sonnet_model["Claude Sonnet model"]
  Claude_Sonnet_model --> HR_chatbot_UI
  EKS_on_Outposts --> Amazon_Bedrock_private_endpoint

Components

Data flow

For ASR, call-center recordings start in an on-premise Hadoop cluster, are split into agent/customer channels, cleaned with voice activity detection, chunked into 2-to-20-second phrases, transcribed by the previous speech-to-text model, transferred to Outposts, filtered by NER to remove sensitive chunks, converted into encrypted Mel spectrograms, synchronized to regional S3, fine-tuned in SageMaker, and returned on premise for inference. For HR RAG, HR policies are embedded locally, stored as vectors in Postgres on Outposts, matched against embedded user prompts, attached as context, sent to Bedrock/Claude in the region, and returned to the on-premise chatbot.

Why this architecture

The hybrid architecture exists because the bank needed cloud flexibility, cloud GPU capacity, and managed AI services, but could not allow keys or sensitive customer data to leave its control. AWS Outposts and XKS let the bank run AWS-managed services locally with locally held keys; the ASR design sends only cleaned, encrypted Mel spectrograms to cloud GPUs; and the HR bot keeps the knowledge base, embeddings, and vector store on Outposts while sending only prompt plus retrieved context to Bedrock.

Models used: custom ASR model · previous-generation speech-to-text model · Named Entity Recognition model · embedding model · Claude Sonnet 4.5 · Amazon Bedrock models

Scale

• BCC serves more than 3 million clients across 20 branches and more than 150 offices.
  “The bank has 20 branch and more than 150 offices. Service over three million clients”transcript, stated
• BCC subscribed to two AWS Outpost racks in Q1 2024.
  “we subscribe for two Outpost racks in Q1 2024.”transcript, stated
• The bank operates more than 20 Kubernetes clusters.
  “these workloads will also execute large, more 20 clusters and we want to explore how we can migrate these workloads using cloud.”transcript, stated
• ASR chunks range from 2 seconds to 20 seconds.
  “each chunk with variable duration from two seconds to 20, depends on the duration of the actual phrase.”transcript, stated
• The custom ASR model improved accuracy by 9% for Russian, 15% for Kazakh, and 23% for mixed language.
  “9% for Russian, it's 15% for Kazakh. And what is the most important, the most significant improvement for the mixed language”transcript, stated

  “it's 23% for the mixed language.”transcript, stated
• The ASR model saves four million Kazakhstani tenge every month.
  “it allows bank to save four million tenge every month.”transcript, stated
• Roughly 70% of HR requests go through the HR bot.
  “now roughly 70% of their HR request goes through this HR-bot, which is quite good.”transcript, stated

Alternatives considered (roads not taken)

Purely local infrastructure for all AI/ML workloads

Rejected because: Growing business needs and local limitations led the bank to scale with cloud resources.

Run all ASR fine-tuning on Outposts

Rejected because: BCC's Outposts did not have GPUs, and local GPUs were sufficient for inference but not large-scale fine-tuning.

Use commercial or open-source ASR models as-is

Rejected because: Available models did not handle Kazakh/Russian mixed language and were trained on 16 kHz data while the bank stored 8 kHz recordings.

Send the full HR knowledge base to the cloud model

Rejected because: The hybrid RAG design keeps HR policies embedded and stored locally, sending only the prompt plus retrieved context to Bedrock.

Patterns

privacy-preserving-processing (governance)

Encryption keys stay in the bank, sensitive chunks are removed before training data leaves the perimeter, and cloud training uses encrypted transformed spectrogram data.

fine-tuning-adaptation (model-adaptation)

The bank fine-tuned a custom ASR model for Kazakh/Russian mixed-language call-center audio using cleaned internal recordings.

rag-retrieval (retrieval)

The HR bot embeds local HR policies, retrieves relevant context from a Postgres vector database on Outposts, and sends only prompt plus context to Bedrock.

cost-aware-routing (operations)

The system uses cloud GPU power for compute-heavy fine-tuning, then returns the model for on-premise inference where the bank has sufficient local GPU capacity.

safety-policy-enforcement (governance)

The architecture is shaped around regulatory requirements for encryption, local key control, anonymization, and confidential-data boundaries.

eval-driven-development (quality)

The ASR use case reports language-specific accuracy improvements and monthly savings, while the HR bot is assessed through request share and employee feedback.

Tradeoffs

Hybrid cloud instead of all-local infrastructure

Chose AWS Outposts plus cloud regions and other clouds over keeping all workloads only in local data centers.

Rationale: The bank needed scalability, innovation, fault tolerance, and economic efficiency while retaining compliance controls.

Cloud fine-tuning with on-premise inference

Chose SageMaker/cloud GPUs for training and local GPU power for inference over trying to fine-tune entirely on local or Outposts GPU capacity.

Rationale: Cloud GPUs solved the fine-tuning bottleneck while inference remained within the bank's controlled environment.

Remove sensitive chunks instead of redacting and retaining them

Chose delete text/audio chunks containing personal or confidential data over attempting to preserve all training examples through redaction.

Rationale: The requirement was that none of the sensitive data leave the perimeter, so entire affected chunks were removed.

Mel spectrograms as cloud training representation

Chose encrypted Mel spectrograms for SageMaker fine-tuning over sending raw audio/text directly to cloud training.

Rationale: Mel spectrograms are an ASR best practice and create a representation that is not raw audio or text after sensitive content has been removed.

Hybrid RAG for HR support

Chose local embeddings/vector store with Bedrock generation over a fully local chatbot or a fully cloud-hosted knowledge base.

Rationale: The bank gains access to Bedrock/Claude while keeping the HR knowledge base and vector search on Outposts.

Gotchas (what bit them)

Regulatory rules determined the architecture before model choices did.

Root cause: Banking requirements required local key storage and customer-data anonymization, so ordinary cloud-native AI pipelines were not acceptable.

Resolution: Deploy AWS Outposts, XKS, double encryption, and on-premise anonymization so cloud services can be used without surrendering key control.

Outposts did not provide the GPU capacity needed for ASR fine-tuning.

Root cause: BCC's Outposts configuration did not include GPUs, and local GPUs were sufficient for inference but not fine-tuning.

Resolution: Use cloud GPUs/SageMaker for fine-tuning, then return artifacts for on-premise inference.

Existing ASR models did not fit the bank's language and audio reality.

Root cause: Kazakh/Russian mixed speech had no market model, and bank recordings were 8 kHz while common ASR datasets were 16 kHz.

Resolution: Build a custom 8 kHz dataset from real call-center data and fine-tune a custom ASR model.

Training data could not leave the perimeter with sensitive content intact.

Root cause: Internal security required that personal, sensitive, and confidential data not leave the bank's security perimeter.

Resolution: Run NER on text files and remove both text and corresponding audio chunks where sensitive data is identified.

Hybrid RAG still sends some employee query context to the cloud.

Root cause: Bedrock runs in the AWS region, so the prompt plus retrieved HR context travels from Outposts to the region for generation.

Resolution: Keep HR knowledge base embeddings/vector storage local and send only the user prompt plus retrieved context through the secured Bedrock path.

Evaluation

The presentation reports quantitative ASR accuracy improvements by language, monthly cost savings versus the previous commercial ASR solution, HR bot request share, and employee feedback. It does not describe a formal chatbot evaluation harness or detailed methodology for the ASR accuracy numbers.

Metrics: 9% ASR accuracy improvement for Russian · 15% ASR accuracy improvement for Kazakh · 23% ASR accuracy improvement for mixed language · four million tenge saved per month · roughly 70% of HR requests go through the HR bot · mostly positive employee feedback

Outcomes

• BCC built a secure hybrid multi-cloud platform that supports regulated AI/ML workloads.
• The custom ASR model improved accuracy for Russian, Kazakh, and mixed-language speech and is used in the post-call analytics pipeline.
• The custom ASR model saves four million tenge per month compared with the previous commercial ASR solution.
• The HR bot handles roughly 70% of HR requests and has mostly positive employee feedback.
• The bank plans to reuse the architecture for other GenAI tasks, including IT support and procurement.

The case provides concrete operational metrics but is an AWS/BCC presentation; accuracy and savings methodology are not independently detailed. The reusable pattern is hybrid compliance architecture: local keys, local anonymization, cloud training/generation where allowed, and on-premise inference or vector storage where required.