Streamlining Background Check Classification with Fine-tuned Small Language Models

checkr · HR · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Background-check adjudication classification

Checkr needed to classify the hardest background-check adjudication records into 230 categories with high accuracy, low latency, and reasonable cost after its prior DNN covered only a small fraction of hard cases.

Constraints:

• The hard 2% of cases involved noisy unstructured text that was difficult for humans and automated models.
• The task was synchronous and needed low-latency SLAs for near-real-time employment insights.
• The classification affects hiring decisions, so accuracy and compliance matter.
• Inference costs had to remain reasonable while processing millions of tokens monthly.

Architecture

Checkr experimented from GPT-4 expert prompting and RAG to fine-tuned small language models, then productionized a Predibase-hosted fine-tuned llama-3-8b-instruct model. The production architecture uses Predibase fine-tuning, LoRA/LoRAX serving, low-latency inference, dashboards for model performance, and multi-LoRA serving to support additional use cases without extra GPUs.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  complex_background_check_records["Complex background-check records"] --> fine_tuned_slm_classifier["Fine-tuned SLM classifier"]
  fine_tuned_slm_classifier --> predibase_fine_tuning_platform["Predibase fine-tuning platform"]
  fine_tuned_slm_classifier --> lorax_serving_layer["LoRAX serving layer"]
  predibase_fine_tuning_platform --> model_tracking_dashboards["Model tracking dashboards"]
  lorax_serving_layer --> classification_output["Classification output"]

Components

Data flow

Complex unstructured background-check records flow into a fine-tuned SLM classifier. The model predicts one of 230 categories for the hard adjudication subset, served through Predibase with LoRAX. Production dashboards and model-version tooling monitor performance, latency, and efficiency, while future LoRA adapters can share the serving infrastructure.

Why this architecture

Checkr moved to fine-tuned SLMs because GPT-4 and RAG performed well on easy cases but lagged on the difficult 2%, while a small domain-adapted model delivered better accuracy, much lower latency, and lower cost for the synchronous production classifier.

Models used: GPT-4 · Llama-2-7b · llama-3-8b-instruct · Llama-2-chat · Predibase LoRA adapters

Scale

• Checkr serves over 100,000 customers.
  “is a technology company that specializes in delivering modern and compliant background checks for over 100,000 customers of all sizes.”article, stated
• The existing automated adjudication solution reduced manual reviews by 95%.
  “Checkr provides an automated adjudication solution that has been shown to reduce manual reviews by 95%.”article, stated
• The hard subset required classifying the remaining 2% of data into 230 categories.
  “The remaining 2%, however, are complex cases that require categorizing data records into 230 distinct categories.”article, stated
• The production model achieved 0.15 second responses, 30x faster than GPT-4 experiments.
  “consistently delivers .15 sec response times for our production traffic which is 30x faster than our GPT-4 experiments.”article, stated
• Checkr reduced inference costs by 5x compared with GPT-4.
  “We reduced our inference costs by 5x compared to GPT-4 by fine-tuning and serving SLMs on Predibase.”article, stated

Alternatives considered (roads not taken)

Previous DNN classifier

Rejected because: It classified only 1% of the difficult 2% with decent accuracy, leaving another 1% unclassified.

GPT-4 expert LLM

Rejected because: GPT-4 achieved only 80-82% accuracy on the hardest 2% and had worse latency/cost than fine-tuned SLMs.

GPT-4 plus RAG over training examples

Rejected because: RAG improved easier cases but hurt difficult cases by steering the LLM away from better reasoning.

Traditional full fine-tuning

Rejected because: Predibase PEFT with LoRA reduced training cost and time while delivering comparable performance.

Patterns

fine-tuning-adaptation (model adaptation)

Checkr fine-tuned small open-source LLMs on its hard adjudication classification task, culminating in a llama-3-8b-instruct production model.

eval-driven-development (model selection)

The team compared GPT-4, Expert+RAG, Llama-2 fine-tuning, combined models, and Predibase-hosted fine-tuning across accuracy, latency, and cost.

latency-optimization (serving)

The production system uses optimized SLM serving to meet synchronous low-latency requirements with 0.15 second response times.

cost-aware-routing (cost optimization)

Fine-tuned SLMs and multi-LoRA serving reduce inference cost and enable additional use cases without adding GPUs.

observability-tracing (operations)

Predibase dashboards let Checkr track model versions, loss graphs, production efficiency, and performance impact over time.

Tradeoffs

Fine-tune small language models for the hardest records.

Chose Llama-based SLMs on Predibase over GPT-4 expert prompting and GPT-4 plus RAG.

Rationale: Fine-tuned SLMs improved difficult-case accuracy while cutting latency and cost for the synchronous production workflow.

Use LoRA/PEFT serving.

Chose Predibase PEFT and LoRAX multi-LoRA serving over Full fine-tuning and separate GPU deployments for each adapter.

Rationale: LoRA reduced training cost/time and enabled additional adapters without extra GPUs.

Keep prompts short after fine-tuning.

Chose Concise prompts over Prompt-engineering-heavy performance tuning.

Rationale: Fine-tuned model performance was largely independent of prompt details, so shorter prompts reduced token usage and cost.

Gotchas (what bit them)

RAG can hurt hard classification cases when examples pull reasoning the wrong way.

Root cause: Training-set examples helped easier cases but misled the LLM on more difficult records.

Resolution: Fine-tune a smaller model directly on the hard classification task instead of relying on retrieved examples.

Difficult adjudication records were not covered by earlier automation.

Root cause: The remaining 2% involved noisy unstructured text and 230 categories, while the original DNN covered only half of that hard slice.

Resolution: Use fine-tuned SLMs to classify the most complex records with higher accuracy and coverage.

Fine-tuned classifiers can appear overconfident.

Root cause: Confidence intervals from log probabilities produced skewed distributions with consistently high confidence scores.

Resolution: Lower temperature and top_k to broaden the confidence distribution while maintaining precision.

Evaluation

Checkr compared multiple model and architecture options on accuracy for easy and difficult datasets, then used production metrics dashboards and convergence/loss monitoring during fine-tuning.

Metrics: GPT-4 expert: 87-88% easy accuracy and 80-82% hard accuracy. · Expert+RAG: 96% easy accuracy but lower hard-case accuracy. · Fine-tuned Llama-2-7b: 97% easy accuracy and 85% difficult accuracy. · Fine-tuned llama-3-8b-instruct: 90% accuracy on the most challenging 2%. · Production latency: 0.15 seconds. · Cost reduction: 5x versus GPT-4.

Outcomes

• The best fine-tuned llama-3-8b-instruct model reached 90% accuracy on the most challenging 2% of cases.
• Predibase production inference delivered 0.15 second response times, 30x faster than GPT-4 experiments.
• Fine-tuned SLM serving reduced inference costs by 5x compared with GPT-4.
• Multi-LoRA serving enables future use-case expansion without more GPUs.

Checkr serves over 100,000 customers, conducts millions of background checks each month, processes 98% of data with logistic regression, and targets the remaining 2% hard cases across 230 categories.