Migration of Credit AI RAG Application from Multi-Cloud to AWS Bedrock

octus · Finance · 2023 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Finance

Octus needed to migrate a production Credit AI RAG product from fragmented multi-cloud infrastructure to a simpler, scalable, compliant architecture.

Constraints:

• production users require low latency and high availability
• financial services customers require SOC2, legal, and governance controls
• multi-cloud tooling added cost and operational complexity
• document sync and embedding needed to scale

Architecture

Octus migrated Credit AI from a split Azure/OpenAI plus AWS estate into an AWS-centered RAG architecture. Ingestion starts when new documents trigger Lambda validation, metadata extraction, and Textract layout/text extraction; extracted content is stored separately in S3 with lineage to source documents, chunked through Bedrock knowledge-base workflows, embedded with Cohere, and indexed in Amazon OpenSearch. The Q&A path runs through a Fargate web app and Amazon MSK orchestration layer, embeds the user query, retrieves chunks from OpenSearch, applies Bedrock Guardrails for content safety, PII, hallucination, and relevancy controls, then generates a response. Supporting data services, Terraform/CI-CD, Datadog/CloudWatch/PagerDuty, and per-client knowledge-base isolation carry the production/compliance load.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  lambda-ingestion["lambda-ingestion"] --> amazon-textract["amazon-textract"]
  amazon-textract --> s3-extracted-content["s3-extracted-content"]
  s3-extracted-content --> bedrock-knowledge-bases["bedrock-knowledge-bases"]
  bedrock-knowledge-bases --> cohere-embedding-model["cohere-embedding-model"]
  cohere-embedding-model --> amazon-opensearch["amazon-opensearch"]
  fargate-web-app["fargate-web-app"] --> msk-orchestration["msk-orchestration"]
  msk-orchestration --> cohere-embedding-model
  msk-orchestration --> amazon-opensearch
  msk-orchestration --> bedrock-guardrails["bedrock-guardrails"]
  bedrock-guardrails --> bedrock-generation["bedrock-generation"]
  unified-data-services["unified-data-services"] --> msk-orchestration
  msk-orchestration --> monitoring-alerting["monitoring-alerting"]

Components

Data flow

For ingestion, a new document triggers Lambda, which validates document type and size, extracts metadata, calls Textract for text/layout/structure, and writes extracted content to a separate S3 prefix with lineage. Bedrock chunks that text, Cohere creates vectors, and OpenSearch stores them for retrieval. For Q&A, the Fargate web app validates the user, MSK coordinates the query cycle, the query is embedded, relevant chunks are fetched from OpenSearch, Bedrock Guardrails checks quality and security, and the model generates the answer back to the user. Data services supply normalized financial identifiers and document APIs, while monitoring and alerting watch the production system.

Why this architecture

Octus wanted to simplify a component-heavy multi-cloud system without losing swapability, compliance, or production scale. Bedrock consolidated managed RAG steps, model access, guardrails, and AWS service integration where most Octus data/services already lived. The architecture pushes undifferentiated RAG infrastructure to managed providers, keeps per-client knowledge bases physically isolated for compliance, and leaves engineering teams focused on customer features instead of GPUs, CUDA errors, cross-cloud data movement, and external safety calls.

Models used: Cohere embedding models · Amazon Bedrock models

Scale

• Credit AI launched in 2023 as a production GenAI product in credit markets.
  “when we launched Credit AI as the first Gen-AI product on credit market in the industry, that was back in 2023.”transcript, stated
• Production requirements included millions of documents, sub-second latency, thousands of concurrent users, and 99%+ accuracy.
  4 supporting quotes

  “but in production, you're talking about millions of documents. Latency. In POC, you will typically be okay with whatever latency you are getting”transcript, stated

  “In production, you're looking at sub-second latency, and you need to have that so that you have, you know, great end-user experience for your customers.”transcript, stated

  “In production, you're talking about thousands of concurrent users.”transcript, stated

  “we have to have 99% plus accuracy, because that's very important for us to have a solid product in the market.”transcript, stated
• Cohere managed embeddings increased throughput by 10x.
  “So implementing this increased their throughput by 10x. And from a bottleneck, it became a non-issue.”transcript, stated
• Octus customers span America, Europe, and Asia and need multilingual embedding support.
  “Their customer base is across America, Europe, and Asia, and the model has multilingual capabilities which they needed.”transcript, stated
• The migration reduced infrastructure cost by 78% and cost per question by 87%.
  “So the cost went down by 78%. We are not managing anymore our own GPUs.”transcript, stated

  “And that also means the cost per question went down by 87%.”transcript, stated
• Document sync improved from hours to minutes.
  “Before, we were able to sync documents in hours, and now we're able to sync documents in minutes just because we can scale those embeddings with Amazon Bedrock.”transcript, stated

Alternatives considered (roads not taken)

Continue operating a split Azure/OpenAI plus AWS architecture

Rejected because: The split required shuttling data between clouds and increased operational complexity and cost.

Fixed-size chunking for the production document corpus

Rejected because: It worked for small documents but performed poorly on long, complex financial documents because chunks could cut mid-sentence, mid-paragraph, or mid-table.

Hierarchical parent/child chunking with larger parent chunks

Rejected because: It produced promising results, but the chosen Cohere embedding model had a 512-token limit that could not handle larger parent chunks.

Self-hosted embedding models on dedicated GPUs

Rejected because: The self-hosted setup was costly, hard to scale, and could take down the whole pipeline when the instance failed.

External third-party hallucination/relevancy checks

Rejected because: TrueLens added latency, cost, and another external call; Bedrock Guardrails put safety and relevancy checks in the pipeline.

Patterns

rag-retrieval (retrieval)

Credit AI uses managed RAG capabilities for ingestion, retrieval, prompt augmentation, and response generation.

privacy-preserving-processing (governance)

The migration had to satisfy SOC2, legal, governance, and financial-compliance concerns.

safety-policy-enforcement (governance)

Bedrock Guardrails are used for content filtering, PII-leakage prevention, hallucination controls, and relevancy checks.

Tradeoffs

Move from self-managed and multi-cloud pieces to managed Bedrock services.

Chose Amazon Bedrock managed services over managing GPU and RAG infrastructure independently across clouds.

Rationale: The migration reduced infrastructure ownership and cost while improving document sync speed.

Use semantic chunking for complex financial documents.

Chose LLM-assisted semantic chunking over fixed chunking and hierarchical parent chunks.

Rationale: Fixed chunks broke long financial documents at bad boundaries, and hierarchical parent chunks hit the Cohere embedding model's token limit; semantic chunking cost more but produced better results.

Physically isolate each client's private knowledge base.

Chose separate knowledge bases per client over shared knowledge base partitioned only by client ID.

Rationale: Financial-services customers and auditors demanded stronger physical data boundaries for private client documents.

Move guardrails into the Bedrock pipeline.

Chose Bedrock Guardrails built into the RAG path over third-party TrueLens calls for hallucination/relevancy checks.

Rationale: In-pipeline guardrails reduced external-call complexity, token usage, latency, and cost.

Gotchas (what bit them)

Production GenAI migration adds requirements that prototypes can ignore.

Root cause: POCs can tolerate small user counts, loose latency, and weaker infrastructure quality; production cannot.

Resolution: Octus defined non-negotiables around sub-second latency, thousands of users, high availability, disaster recovery, accuracy, and compliance.

Evaluation

The case reports operational results after migration, including cost and sync-time improvements.

Metrics: infrastructure cost · cost per question · document sync time · latency · accuracy

Outcomes

• Infrastructure cost fell 78%, cost per question fell 87%, and document sync moved from hours to minutes.

The source frames the system as production for sophisticated financial-services customers.