Fine-tuning Multimodal Models for Banking Document Processing

apoidea_group · Finance · 2025 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: banking document processing

Apoidea and AWS needed to improve visual information extraction from complex banking documents, especially table structure recognition in financial statements, because traditional modular OCR/layout/entity pipelines were complex, error-prone, and difficult to scale in regulated banking environments.

Constraints:

banking documents contained sensitive financial data requiring banking-grade security and compliance controls
documents varied across institutions, layouts, scan quality, merged cells, and multi-page structures
existing modular pipelines increased system complexity and propagated upstream errors downstream
training data for clean document-understanding models was scarce and imbalanced for complex forms and long tables
the solution needed scalable distributed training and efficient high-throughput inference

Architecture

AWS and Apoidea fine-tuned Qwen2-VL-7B-Instruct for banking table structure recognition using LLaMA-Factory on SageMaker HyperPod. The pipeline preprocesses document images into HTML table ground truth, uses Slurm sbatch to orchestrate QLoRA data-parallel distributed training on HyperPod, then serves a 4-bit quantized model with vLLM behind RESTful APIs for integration with document-processing pipelines. Evaluation used FinTabNet financial tables and TEDS/TEDS-S metrics to compare the fine-tuned model against Claude baselines and the Qwen2-VL base model.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  banking-document-images[Banking document images]
  image-to-html-preprocessing[Image-to-HTML preprocessing]
  llama-factory-fine-tuning[LLaMA-Factory fine-tuning]
  sagemaker-hyperpod-training-cluster[SageMaker HyperPod training cluster]
  slurm-job-scheduler[Slurm job scheduler]
  qwen2-vl-7b-instruct-model[Qwen2-VL-7B-Instruct model]
  vllm-inference-server[vLLM inference server]
  document-processing-pipeline[Document processing pipeline]
  teds-evaluation-suite[TEDS evaluation suite]
  financial-services-security-controls[Financial-services security controls]
  banking-document-images --> image-to-html-preprocessing
  image-to-html-preprocessing --> llama-factory-fine-tuning
  llama-factory-fine-tuning --> sagemaker-hyperpod-training-cluster
  sagemaker-hyperpod-training-cluster --> slurm-job-scheduler
  slurm-job-scheduler --> llama-factory-fine-tuning
  llama-factory-fine-tuning --> qwen2-vl-7b-instruct-model
  qwen2-vl-7b-instruct-model --> vllm-inference-server
  qwen2-vl-7b-instruct-model --> teds-evaluation-suite
  vllm-inference-server --> document-processing-pipeline
  financial-services-security-controls --> sagemaker-hyperpod-training-cluster
  financial-services-security-controls --> vllm-inference-server

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Banking document images`	Bank statements, KYC documents, financial statements, and complex financial tables to be processed.	`Image-to-HTML preprocessing`
`Image-to-HTML preprocessing`	Creates image inputs and HTML table outputs used as supervised fine-tuning examples.	`LLaMA-Factory fine-tuning`
`LLaMA-Factory fine-tuning`	Runs efficient QLoRA-based fine-tuning for Qwen2-VL-7B-Instruct.	`Qwen2-VL-7B-Instruct model`, `SageMaker HyperPod training cluster`
`SageMaker HyperPod training cluster`	Provides distributed training infrastructure, shared file system integration, scalable GPU resources, and managed ML workflow support.	`Slurm job scheduler`, `LLaMA-Factory fine-tuning`
`Slurm job scheduler`	Orchestrates distributed training scripts and resource scheduling for the fine-tuning job.	`SageMaker HyperPod training cluster`, `LLaMA-Factory fine-tuning`
`Qwen2-VL-7B-Instruct model`	Vision-language model adapted for banking table structure recognition.	`vLLM inference server`
`vLLM inference server`	Hosts the 4-bit quantized model with efficient memory management, optimized attention, batching, and REST API integration.	`Document processing pipeline`, `Financial-services security controls`
`Document processing pipeline`	Consumes structured table outputs for downstream banking document automation.	`vLLM inference server`
`TEDS evaluation suite`	Compares generated HTML table structure and content with ground truth.	`Qwen2-VL-7B-Instruct model`
`Financial-services security controls`	Applies encryption, private-networking, IAM, API Gateway, and authentication safeguards around sensitive financial data.	`SageMaker HyperPod training cluster`, `vLLM inference server`

Data flow

Banking document images and HTML table outputs are prepared as training examples so the model learns table layout and expected output structure. LLaMA-Factory fine-tunes Qwen2-VL-7B-Instruct with QLoRA and distributed data parallel training on SageMaker HyperPod, orchestrated by Slurm. The resulting model is quantized to 4-bit, hosted with vLLM for memory-efficient high-throughput inference, exposed through REST APIs, and evaluated by comparing generated HTML tables with ground truth using TEDS and TEDS-S.

“We start the data preprocessing using the image input and HTML output. We choose the HTML structure as the output format because it is the most common format for representing tabular data in web applications.”article, stated

“We then use LLaMA-Factory to fine-tune the Qwen2-VL-7B-Instruct model on the preprocessed data.”article, stated

Why this architecture

The architecture uses a fine-tuned multimodal model because banking documents combine layout, text, and visual structure, while fragmented OCR/layout/entity pipelines lose information and propagate upstream errors. LLaMA-Factory and HyperPod make domain fine-tuning practical, and vLLM plus 4-bit quantization support production integration.

“Unlike traditional information extraction systems that rely on fragmented processing pipelines, these models can simultaneously analyze document layouts, extract text content, and interpret visual elements.”article, stated

“During inference, we use vLLM for hosting the quantized model, which provides efficient memory management and optimized attention mechanisms for high-throughput inference.”article, stated

Models used: Qwen2-VL-7B-Instruct · Anthropic Claude 3 Haiku · Anthropic Claude 3.5 Sonnet

Scale

SuperAcc had been deployed and trusted by more than 10 financial-services clients.
“SuperAcc has been successfully deployed and trusted by over 10 financial services industry (FSI) clients”article, stated
Financial spreading time reportedly dropped from 4-6 hours to 10 minutes plus less than 30 minutes of staff review.
“the financial spreading process, which previously required 4–6 hours, can now be completed in just 10 minutes, with staff needing less than 30 minutes to review the results.”article, stated
SME bank-statement review across six months of statements reportedly dropped to 10 minutes.
“the review process for multiple bank statements spanning 6 months—used to extract critical data such as sales turnover and interbank transactions—has been reduced to just 10 minutes.”article, stated
The article reports over 80% ROI from SuperAcc banking deployments.
“These advancements have led to an impressive return on investment (ROI) of over 80%”article, stated
The multimodal model was a 7B-parameter Qwen2-VL model.
“fine-tuning the Qwen2-VL-7B-Instruct model using LLaMA-Factory on SageMaker HyperPod.”article, stated
Domain-specific fine-tuning achieved a five-point TEDS improvement with only 10% of the data compared with general datasets.
“Using domain-specific data, we achieved a 5-point improvement in TEDS score with only 10% of the data compared to using general datasets.”article, stated

Alternatives considered (roads not taken)

traditional modular OCR/layout/entity pipelines

Rejected because: they increased complexity, failed to use all information jointly, and propagated upstream errors.

“Current solutions rely on orchestrating models for tasks such as layout analysis, entity extraction, and table structure recognition.”article, stated

single-modality document models

Rejected because: they could not jointly reason over text, visual layout, and spatial relationships.

“single-modality models fail to use the multi-faceted nature of information, potentially leading to less precise and accurate predictions.”article, stated

base Qwen2-VL-7B-Instruct without fine-tuning

Rejected because: the base model scored far below the fine-tuned model on TEDS and TEDS-S for the table-recognition task.

"Qwen2-VL-7B-Instruct (Base)

23.4

25.3

Qwen2-VL-7B-Instruct (Fine-tuned)

81.1

89.7"

— article, stated

large uncustomized commercial multimodal models

Rejected because: Claude 3 Haiku was below the fine-tuned model, while Claude 3.5 Sonnet was approached by the smaller fine-tuned model with more efficient compute requirements.

“The fine-tuned version not only surpassed the performance of Anthropic’s Claude 3 Haiku, but also approached the accuracy levels of Anthropic’s Claude 3.5 Sonnet”article, stated

Patterns

`fine-tuning-adaptation` (model-operations)

Apoidea and AWS fine-tuned Qwen2-VL-7B-Instruct on domain-specific banking document table data using LLaMA-Factory and QLoRA.

“We then use LLaMA-Factory to fine-tune the Qwen2-VL-7B-Instruct model on the preprocessed data.”article, stated

`structured-output-contract` (output-control)

The model was trained to emit HTML table structures from document images so outputs could be parsed, rendered, reviewed, or modified.

“We choose the HTML structure as the output format because it is the most common format for representing tabular data in web applications.”article, stated

`eval-driven-development` (evaluation)

Model quality was measured with TEDS and TEDS-S against ground-truth HTML tables from the FinTabNet evaluation dataset.

“For quantitative evaluation, we employed the Tree Edit Distance-based Similarity (TEDS) metric, which assesses both structural and content similarity between generated HTML tables and ground truth.”article, stated

`latency-optimization` (serving)

The inference path uses 4-bit quantization and vLLM memory/attention optimizations for high-throughput document processing.

“The deployment process involves applying 4-bit quantization to reduce model size while maintaining accuracy, configuring the vLLM server with optimal parameters for batch processing and memory allocation”article, stated

`safety-policy-enforcement` (governance)

The banking document pipeline includes encryption, private networking, IAM roles, API Gateway protections, and authentication for sensitive financial data.

“robust security measures must be incorporated throughout the ML lifecycle.”article, stated

Tradeoffs

Use a unified multimodal model for document understanding.

Chose fine-tuned Qwen2-VL-7B-Instruct over fragmented layout, OCR, entity, and table-recognition model pipelines.

Rationale: A unified model can jointly use visual layout, text, and spatial relationships while reducing error propagation.

“By integrating visual and textual understanding into a unified framework, multimodal models offer a transformative approach to document analysis.”article, stated

Use HTML as the target output format.

Chose HTML table structure over an unspecified proprietary or unstructured output.

Rationale: HTML is common, parseable, renderable in browsers, and usable for manual review or modification.

“It is straightforward to parse and visualize, and it is compatible with most web browsers for rendering on the website for manual review or modification if needed.”article, stated

Use LLaMA-Factory with QLoRA.

Chose parameter-efficient multimodal fine-tuning over heavier full-parameter training as the default path.

Rationale: The framework reduces compute and memory needs and lets users fine-tune models with limited hardware.

“It significantly reduces the computational and memory requirements for fine-tuning large models by using techniques like LoRA and quantization”article, stated

Host with vLLM and 4-bit quantization.

Chose quantized high-throughput REST-serving path over unoptimized model serving.

Rationale: The deployment needed efficient memory management, optimized attention, batching, and integration with existing document pipelines.

“exposing the model through RESTful APIs for quick integration with existing document processing pipelines.”article, stated

Gotchas (what bit them)

Traditional modular document pipelines increased complexity and error propagation.

Root cause: Separate layout analysis, entity extraction, and table-recognition models depend on upstream outputs and do not use visual/textual information jointly.

Resolution: The team used a multimodal vision-language model that can jointly analyze layouts, text content, and visual elements.

“downstream models that heavily depend on upstream model outputs are susceptible to error propagation, potentially compounding inaccuracies introduced in earlier stages of processing.”article, stated

Banking document variety and scan quality make robust extraction difficult.

Root cause: Bank statements use different layouts and columns, while scans can be low-quality, poorly aligned, blurry, or faded.

Resolution: Domain-specific fine-tuning adapts the multimodal model to specialized document layouts and terminology.

“bank statement formats vary significantly across financial institutions, with each bank using unique layouts, different columns, transaction descriptions”article, stated

Fine-tuning data imbalance can bias performance.

Root cause: Datasets lacking complex elements such as long tables and merged cells underrepresent hard document formats.

Resolution: The article recommends maintaining a balanced distribution of document types and combining real and synthetic data when annotated data is limited.

“imbalanced datasets, particularly those lacking sufficient examples of complex elements like long tables and forms with merged cells, can lead to biased performance.”article, stated

Sensitive financial documents impose security requirements across training and serving.

Root cause: Financial services data must remain protected through the whole ML lifecycle.

Resolution: The recommended setup uses encryption, VPC endpoints/private subnets, least-privilege IAM, API Gateway protections, and token authentication.

“Secure model hosting with vLLM requires deployment in private VPC subnets with proper Amazon API Gateway protections and token-based authentication.”article, stated

Evaluation

The team evaluated table-structure recognition on FinTabNet, comparing generated HTML with ground truth using TEDS for structural/content similarity and TEDS-S for structural similarity.

Metrics: Claude 3 Haiku TEDS 69.9 and TEDS-S 76.2 · Claude 3.5 Sonnet TEDS 86.4 and TEDS-S 87.1 · Qwen2-VL-7B-Instruct base TEDS 23.4 and TEDS-S 25.3 · Qwen2-VL-7B-Instruct fine-tuned TEDS 81.1 and TEDS-S 89.7 · 5-point TEDS improvement with 10% of the data when using domain-specific data

“Our evaluation focused on the FinTabNet dataset, which contains complex tables from S&P 500 annual reports.”article, stated

“Qwen2-VL-7B-Instruct (Fine-tuned) 81.1 89.7 The evaluation results reveal significant advancements in our fine-tuned model’s performance.”article, stated

Outcomes

The fine-tuned Qwen2-VL-7B-Instruct model reached TEDS 81.1 and TEDS-S 89.7 on FinTabNet.
Fine-tuning improved the Qwen2-VL base model from TEDS 23.4 to 81.1.
The fine-tuned model surpassed Claude 3 Haiku on the reported TEDS table and approached Claude 3.5 Sonnet accuracy.
Domain-specific fine-tuning produced a five-point TEDS improvement using only 10% of the data compared with general datasets.
The broader SuperAcc context reports banking workflow reductions from hours to minutes and ROI over 80%.

The system targets high-volume financial document processing in banking, with SuperAcc already deployed at more than 10 FSI clients and the fine-tuned model evaluated on complex S&P 500 annual-report tables.