Streamlining Corporate Audits with GenAI-Powered Document Processing

hapag-lloyd · Other · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Corporate audit documentation and report generation

Hapag-Lloyd needed to reduce manual audit documentation effort, inconsistent report quality, and slow access to audit process documentation while operating across a large global shipping organization.

Constraints:

• Audit finding generation and executive summaries involved manual steps and documentation inconsistencies.
• Auditors needed to query numerous process-documentation files quickly.
• Existing vector database and AWS SysOps setup made fast AI-model setup difficult.
• Audit outputs still needed high quality and consistent evaluation.

Architecture

Hapag-Lloyd built two Databricks GenAI prototypes: a Finding Generation Interface using fine-tuned DBRX to turn audit bullet points into findings and summaries, and a RAG chatbot using Gradio plus Databricks Model Serving to query process documentation. The architecture uses Databricks MLflow for prompt/model evaluation, Delta tables for generated outputs, and the Databricks Data Intelligence Platform for model deployment and audit-data management.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  audit_bullet_points["Audit bullet points"] --> finding_generation_interface["Finding Generation Interface"]
  finding_generation_interface --> fine_tuned_dbrx_model["Fine-tuned DBRX model"]
  fine_tuned_dbrx_model --> mlflow_evaluation["MLflow evaluation"]
  finding_generation_interface --> delta_table["Delta table"]
  audit_documentation_corpus["Audit documentation corpus"] --> rag_chatbot["RAG chatbot"]
  rag_chatbot --> databricks_model_serving["Databricks Model Serving"]

Components

Data flow

Audit data is ingested and prepared, prompts are engineered, DBRX generates findings and summaries, MLflow automates prompt/model evaluation, and generated results are stored in a Delta table. Separately, auditors ask natural-language questions through a Gradio chatbot, which uses RAG and Databricks Model Serving to retrieve contextually relevant answers from audit documentation.

Why this architecture

Hapag-Lloyd used Databricks because its existing infrastructure slowed model setup and deployment. Databricks provided model evaluation, lifecycle management, model serving, and easier instance setup, letting audit teams focus on analysis while prototypes automated repetitive writing and documentation lookup.

Models used: DBRX · Llama 2 70b · Mixtral

Scale

• Hapag-Lloyd operates over 400 offices in 140 countries and a fleet of 280 ships.
  “operations spanning over 400 offices in 140 countries and a fleet of 280 modern ships transporting 11.9 million TEUs annually”summary, stated
• Creating a new finding dropped from 15 minutes to 5 minutes.
  “Auditors now spend only 5 minutes creating a new finding from bullet points, down from 15 minutes.”article, stated
• Executive summary review time decreased from 30 minutes to 7 minutes.
  “The time required to review each executive summary has decreased by 77%, from 30 minutes to just 7 minutes.”article, stated
• There are on average seven findings per audit.
  “a huge time savings given that there are, on average, seven findings per audit.”article, stated

Alternatives considered (roads not taken)

Manual report writing and documentation lookup

Rejected because: Manual methods were time-consuming, inefficient, and inconsistent.

Existing vector databases and AWS SysOps setup

Rejected because: The setup did not support rapid AI-model setup and instance access.

Llama 2 70B and Mixtral for finding generation

Rejected because: DBRX returned better results for the audit finding-generation use case.

Patterns

fine-tuning-adaptation (model adaptation)

Hapag-Lloyd fine-tuned Databricks DBRX for audit finding generation after comparing candidate models.

rag-retrieval (document QA)

The audit chatbot uses retrieval augmented generation to answer natural-language questions over process documentation.

eval-driven-development (model lifecycle)

MLflow automated prompt and model evaluation, reducing manual review and supporting continuous improvement.

tool-use-orchestration (workflow automation)

Databricks orchestrates ingestion, preparation, prompt engineering, evaluation, deployment, model serving, and Delta-table storage for audit workflows.

Tradeoffs

Use DBRX for finding generation.

Chose Fine-tuned DBRX over Llama 2 70B and Mixtral.

Rationale: Hapag-Lloyd evaluated multiple models and found DBRX returned better results for the audit finding-generation prototype.

Add a RAG chatbot for documentation lookup.

Chose Natural-language document querying over Manual search through numerous audit files.

Rationale: The chatbot reduced time spent searching for precise information from many process documents.

Use Databricks as the GenAI platform.

Chose Databricks Data Intelligence Platform with MLflow and Model Serving over Existing AWS SysOps/vector-database setup.

Rationale: Databricks made setup leaner, automated lifecycle steps, and reduced manual model-review work.

Gotchas (what bit them)

Manual audit writing caused inconsistent documentation quality.

Root cause: Traditional report generation involved numerous manual steps.

Resolution: Use a GenAI Finding Generation Interface to draft findings and summaries from bullet points.

Existing infrastructure slowed prototype deployment.

Root cause: Vector databases and AWS SysOps accounts did not support rapid AI-model setup or easy instance access.

Resolution: Deploy GenAI initiatives on the Databricks Data Intelligence Platform.

Evaluation

Hapag-Lloyd used MLflow for automated prompt/model evaluation and reported process-time reductions for finding creation and executive summary review.

Metrics: Finding creation time: 15 minutes to 5 minutes. · Review time per finding: 66% decrease. · Executive summary review time: 30 minutes to 7 minutes. · Executive summary review reduction: 77%.

Outcomes

• Hapag-Lloyd reduced finding-creation time by 66%.
• Executive summary review time fell by 77%.
• Auditors can query process documentation through a natural-language RAG chatbot.
• The prototypes opened a path to further audit automation.

The source reports average seven findings per audit and global company context across over 400 offices, 140 countries, 280 ships, and 11.9 million TEUs annually.