Fine-tuning Mistral 7B for Multilingual Defense Intelligence Sentiment Analysis

vannevar_labs · Government · 2024 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: defense intelligence sentiment analysis

Vannevar Labs needed more accurate and efficient multilingual sentiment analysis for news, blogs, and social media tied to strategic narratives, because GPT-4 prompting was expensive, only reached about 65% accuracy, and struggled with lower-resourced languages such as Tagalog.

Constraints:

• the model had to support mission-focused defense users across U.S. Department of Defense branches
• inputs included multilingual data in Tagalog, Spanish, Russian, and Mandarin
• the application needed low real-time latency
• GPU resources were in short supply for fine-tuning
• the team needed infrastructure for ingestion, training, monitoring, conversion, and production export

Architecture

Vannevar Labs used Databricks to build an end-to-end compound AI system for data ingestion, model fine-tuning, and deployment. They fine-tuned an open-source Mistral 7B model with domain-specific data, targeted single-NVIDIA-A10 deployment for low latency, used MCLI and Python SDK tooling to orchestrate GPU nodes and container images, connected securely to datasets for ingestion, monitored training with third-party tooling, then converted trained models to Hugging Face format and exported them to S3 or the Hugging Face Model Repository for production use.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  multilingual-narrative-data-sources[Multilingual narrative data sources]
  databricks-ingestion-and-training-workspace[Databricks ingestion and training workspace]
  mcli-and-python-sdk[MCLI and Python SDK]
  mistral-7b-fine-tuning-job[Mistral 7B fine-tuning job]
  monitoring-tools[Monitoring tools]
  hugging-face-conversion-and-export[Hugging Face conversion and export]
  production-sentiment-model[Production sentiment model]
  multilingual-narrative-data-sources --> databricks-ingestion-and-training-workspace
  databricks-ingestion-and-training-workspace --> mcli-and-python-sdk
  databricks-ingestion-and-training-workspace --> mistral-7b-fine-tuning-job
  databricks-ingestion-and-training-workspace --> monitoring-tools
  mcli-and-python-sdk --> mistral-7b-fine-tuning-job
  mistral-7b-fine-tuning-job --> monitoring-tools
  mistral-7b-fine-tuning-job --> hugging-face-conversion-and-export
  hugging-face-conversion-and-export --> production-sentiment-model
  production-sentiment-model --> multilingual-narrative-data-sources

Components

Data flow

Publicly available multilingual narrative data is found, aggregated, and ingested into Databricks. The team uses MCLI and the Python SDK to manage GPU nodes, container images, training parameters, and dataset access, then fine-tunes Mistral 7B on domain-specific labels. Training runs can hook into monitoring tools such as Weights & Biases. After training, the model is converted to standard Hugging Face format and exported to S3 or Hugging Face for production deployment.

Why this architecture

The team selected a fine-tuned smaller open-source model because GPT-4 prompting was too expensive and inaccurate for the multilingual task, while Mistral 7B could run on a single A10 GPU with lower real-time latency and be trained on domain-specific data inside Databricks.

Models used: Mistral 7B · GPT-4

Scale

• Vannevar supports hundreds of mission-focused users across U.S. Department of Defense branches.
  “the company leverages advanced software and hardware to support hundreds of mission-focused users across various branches of the U.S. Department of Defense (DoD).”article, stated
• The sentiment task covered multilingual data including Tagalog, Spanish, Russian, and Mandarin.
  “We have data in Tagalog, Spanish, Russian and Mandarin, and GPT-4 struggled with lower-resourced languages like Tagalog.”article, stated
• The target deployment needed to fit on a single NVIDIA A10 GPU.
  “This model was chosen for its open source nature and its ability to efficiently operate on a single NVIDIA A10 Tensor Core GPU.”article, stated
• The team deployed a fully functional fine-tuned sentiment model within two weeks.
  “Within just 2 weeks, we were able to go from a tutorial to deploying a fully functional, fine-tuned sentiment analysis model.”article, stated
• The final model achieved 76% F1, improved from the 65% GPT-4 baseline, and reduced latency by 75%.
  “The fine-tuned model achieved an overall F1 score of 76%, an improvement over the 65% accuracy previously achieved with GPT-4”article, stated

Alternatives considered (roads not taken)

GPT-4 with prompt engineering

Rejected because: it reached only about 65% accuracy, was too expensive, and struggled with lower-resourced languages.

larger or less deployable models

Rejected because: the application needed a model small enough for single-A10 deployment and low real-time latency.

manage fine-tuning infrastructure manually

Rejected because: GPU scarcity and orchestration overhead made infrastructure management part of the bottleneck.

Patterns

fine-tuning-adaptation (model-operations)

Vannevar fine-tuned Mistral 7B on domain-specific multilingual sentiment data to outperform GPT-4 prompting for its defense-intelligence task.

latency-optimization (serving)

The model choice and deployment target were optimized for a single A10 GPU and low real-time latency.

observability-tracing (operations)

Training orchestration used MCLI-managed configurations with third-party monitoring hooks such as Weights & Biases.

cost-aware-routing (serving)

The team replaced an expensive GPT-4 prompting path with a fine-tuned smaller model that was faster and more cost-effective.

Tradeoffs

Fine-tune Mistral 7B instead of continuing GPT-4 prompting.

Chose domain-adapted open-source smaller model over GPT-4 prompt engineering.

Rationale: GPT-4 was too expensive and inaccurate for the multilingual task, while fine-tuned Mistral improved accuracy, cost, and speed.

Target a single-A10 deployment footprint.

Chose Mistral 7B sized for one NVIDIA A10 over larger models that would require heavier serving infrastructure.

Rationale: Real-time mission applications needed low latency and practical GPU deployment.

Use Databricks tooling for model lifecycle orchestration.

Chose MCLI, Python SDK, managed YAML configurations, and platform export over manual GPU/container orchestration.

Rationale: Infrastructure management affected iteration speed, and managed tooling simplified training parameters, GPU scaling, monitoring, and export.

Gotchas (what bit them)

GPT-4 prompting did not meet task accuracy or cost needs.

Root cause: The sentiment task involved multilingual defense-domain data, including lower-resourced languages where GPT-4 struggled.

Resolution: Vannevar fine-tuned Mistral 7B on domain-specific data through Databricks.

Fine-tuning cycles were constrained by GPU access and infrastructure management.

Root cause: The team had to spin up GPU resources and manage training infrastructure while GPUs were in short supply.

Resolution: Databricks MCLI and managed configurations simplified orchestration, scaling, monitoring, and multi-GPU training.

Instruction labels were a companywide bottleneck.

Root cause: The fine-tuning approach required enough domain-specific instruction labels for the task.

Resolution: The customer story does not describe a full labeling-system solution, but the Databricks workflow shortened training/deployment iteration once data was available.

Evaluation

The case compares the Databricks-trained fine-tuned model with the previous GPT-4 prompting baseline across accuracy/F1, cost, and latency for multilingual sentiment analysis.

Metrics: GPT-4 prompt baseline around 65% accuracy · fine-tuned model overall F1 score of 76% · 75% latency reduction versus previous implementations · fully functional model deployed within two weeks

Outcomes

• Vannevar deployed a fully functional fine-tuned sentiment analysis model within two weeks.
• The fine-tuned model reached 76% overall F1 versus a prior GPT-4 baseline around 65% accuracy.
• Latency dropped by 75% compared with previous implementations.
• The Databricks workflow reportedly improved accuracy, cost, and speed.
• The solution enabled larger backfill jobs and more efficient processing of multilingual intelligence data.

The system supports defense mission applications for hundreds of users across DoD branches, processing multilingual public narrative data and enabling large backfill jobs.