Building a Custom Vision LLM for Document Processing at Scale

grab · Tech · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Southeast Asia document OCR and KIE

Grab needed accurate document OCR and key-information extraction for Southeast Asian eKYC and document-processing flows, but traditional OCR, proprietary vision LLMs, and open-source vision LLMs all failed different production constraints.

Constraints:

• documents vary across Southeast Asian languages and formats
• traditional OCR struggled with template diversity
• proprietary LLMs had errors, hallucinations, high latency, and weak Southeast Asian language understanding
• open-source vision LLMs were efficient but not accurate enough for production
• training needed high-quality document labels and GPU-efficient models

Architecture

Grab built a custom vision LLM pipeline around synthetic OCR data, the internal Documint labeling framework, full-parameter fine-tuning, and a final lightweight 1B vision-language model. Documint generates detection, orientation, OCR, and KIE labels from collected documents with human review. The final model combines the Qwen2-VL 2B vision encoder, Qwen2.5 0.5B language decoder, an adjusted projector, and a four-stage training process for SEA document processing.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  document-inputs[Document inputs] --> documint[Documint]
  documint --> human-reviewer[Human reviewer]
  human-reviewer --> curated-document-dataset[Curated document dataset]
  documint --> curated-document-dataset
  curated-document-dataset --> qwen2-vl-2b-baseline[Qwen2-VL 2B baseline]
  qwen2-vl-2b-baseline --> custom-1b-vision-llm[Custom 1B Vision LLM]
  synthetic-ocr-data-pipeline[Synthetic OCR data pipeline] --> custom-1b-vision-llm[Custom 1B Vision LLM]
  vision-encoder[Vision encoder] --> adjusted-projector[Adjusted projector]
  adjusted-projector --> language-decoder[Language decoder]
  vision-encoder --> custom-1b-vision-llm
  adjusted-projector --> custom-1b-vision-llm
  language-decoder --> custom-1b-vision-llm
  custom-1b-vision-llm --> document-processing-workflow[Document processing workflow]

Components

Data flow

Document images and synthetic SEA text images feed training and labeling pipelines. Documint detects document regions, corrects orientation, extracts unstructured OCR text, and produces KIE JSON labels. Grab first fine-tuned Qwen2-VL 2B, then built a 1B model by aligning the projector, enhancing the vision tower, training on language-specific visual data, and full-parameter fine-tuning on curated document data.

Why this architecture

Grab chose a specialized small vision LLM because document processing needed SEA-script accuracy and production latency. Full fine-tuning taught visual patterns that LoRA missed, while the custom 1B architecture kept the strong Qwen2-VL vision encoder but reduced deployment cost and latency with a compact decoder and staged training.

Models used: Qwen2-VL 2B · Qwen2.5 0.5B · miniCPM · Llama3.2 Vision · Pixtral 12B · GOT-OCR2.0 · NVLM 1.0 · ChatGPT · Gemini

Scale

• Grab's document challenge spans Southeast Asia languages and document formats.
  “This task is especially challenging in Southeast Asia (SEA) due to the diversity of languages and document formats.”article, stated
• The selected base model was Qwen2-VL 2B.
  “Our exploration of open-source options—including Qwen2VL, miniCPM, Llama3.2 Vision, Pixtral 12B, GOT-OCR2.0, and NVLM 1.0—led us to select Qwen2-VL 2B as our base multimodal LLM. This decision was driven by several critical factors:”article, stated
• Full fine-tuning improved Thai document accuracy by 70 percentage points and Vietnamese by 40 percentage points.
  “Thai document accuracy increased +70pp from baseline. * Vietnamese document accuracy rose +40pp from baseline.”article, stated
• The custom lightweight model is about 1B parameters.
  “While the Qwen2VL-2B model was a success, the full fine-tuning pushed the limits of GPUs. To optimize resources used and to create a model perfectly tailored to our needs, we decided to build a lightweight Vision LLM (~1B parameters) from scratch.”article, stated
• The 1B model stayed within a 3 percentage-point accuracy gap across most document types.
  “It achieved performance comparable to the larger 2B model, staying within a 3pp accuracy gap across most document types.”article, stated
• External API P99 latency can be 3 to 4 times P50 latency, which Grab found unacceptable for large-scale rollouts.
  “the P99 latency, which can easily be 3 to 4x the P50 latency, which would not be acceptable for Grab’s large scale rollouts.”article, stated
• Grab operates in over 800 cities across eight Southeast Asian countries.
  “Serving over 800 cities in eight Southeast Asian countries, Grab enables millions of people everyday to order food or groceries, send packages, hail a ride or taxi, pay for online purchases or access services such as lending and insurance, all through a single app.”article, stated

Alternatives considered (roads not taken)

Traditional OCR systems

Rejected because: Traditional OCR struggled with the diversity of document templates.

Proprietary Vision LLM APIs

Rejected because: They had SEA-language weaknesses, hallucinations, errors, and unacceptable latency.

LoRA fine-tuning only

Rejected because: LoRA improved Latin-script documents but still struggled with Thai, Vietnamese, and dense unstructured layouts.

Productionizing the fine-tuned 2B model unchanged

Rejected because: The 2B model pushed GPU limits, so Grab built a smaller 1B model optimized for training and deployment.

Patterns

fine-tuning-adaptation (model-ops)

Grab moved from LoRA to full-parameter fine-tuning and then trained a custom 1B vision LLM for its document domain.

synthetic-data-generation (data)

Grab rendered SEA-language text into synthetic images with varied fonts, backgrounds, and augmentations.

data-quality-loop (data)

Documint-generated labels were refined by human reviewers to improve label accuracy.

latency-optimization (operations)

The custom 1B model was built to reduce latency relative to the larger 2B model, traditional OCR, and external APIs.

Tradeoffs

Adaptation method

Chose full-parameter fine-tuning over LoRA-only fine-tuning.

Rationale: Full fine-tuning delivered the non-Latin OCR gains that LoRA did not.

Model architecture

Chose custom 1B model over serving the larger Qwen2-VL 2B model as-is.

Rationale: The 1B model kept comparable accuracy while improving latency and resource use.

Image handling

Chose dynamic/native image resolution over fixed-size resized image inputs.

Rationale: Native resolution preserves text characters for OCR.

Label generation

Chose auto-labeling plus human review over raw automated labels alone.

Rationale: Human review improved the accuracy of labels used to train the specialized model.

Gotchas (what bit them)

Open-source multimodal models lacked enough Southeast Asian visual-text coverage.

Root cause: Their language decoders may be multilingual, but their vision encoders and joint training underrepresent SEA scripts.

Resolution: Train vision components on synthetic SEA OCR data and fully fine-tune on task-specific documents.

LoRA fine-tuning did not solve non-Latin and dense-layout documents.

Root cause: Lightweight parameter updates were not enough for specialized visual OCR/KIE behavior.

Resolution: Use full-parameter fine-tuning and later staged training for the custom 1B model.

External APIs had unacceptable tail latency for large-scale rollout.

Root cause: P99 latency could be several times P50 latency.

Resolution: Build and deploy a lightweight specialized model.

Evaluation

Grab benchmarked candidate vision LLMs on Grab document data, compared LoRA and full fine-tuning, reported field-level OCR/KIE accuracy by document type, and compared latency against 2B, traditional OCR, and external APIs.

Metrics: OCR field-level accuracy · Thai accuracy +70 percentage points · Vietnamese accuracy +40 percentage points · within 3 percentage-point accuracy gap across most document types · P50 latency · P99 latency

Outcomes

• Full fine-tuning improved Thai accuracy by 70 percentage points and Vietnamese accuracy by 40 percentage points from baseline.
• The custom 1B model stayed within 3 percentage points of the larger 2B model across most document types.
• The custom model outperformed the 2B model, traditional OCR models, and external APIs on latency.
• Grab is extending the system across additional Southeast Asian markets.

The case reports 70pp Thai and 40pp Vietnamese accuracy improvements, a ~1B parameter custom model, and expansion across Grab's eight-country Southeast Asia footprint.