Healthcare NLP Pipeline for HIPAA-Compliant Patient Data De-identification

dandelion_health · Healthcare · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: healthcare privacy-preserving data pipelines

Dandelion Health needed to de-identify free-text patient notes for HIPAA compliance while preserving the clinical fidelity needed by researchers and AI model builders.

Constraints:

• under-redaction creates HIPAA re-identification risk
• over-redaction removes clinically valuable research signal
• free-text notes contain unpredictable formatting, headers, footers, embedded metadata, and ASCII tables
• the pipeline runs inside hospital AWS accounts with no internet access
• the de-identification approach must support expert-determination risk measurement rather than only blanket identifier removal
• PHI detection quality must be validated by clinically trained human reviewers
• false positives can leak the masking mechanism when date jitter or HIPS dictionaries are applied incorrectly

Architecture

Dandelion built an air-gapped AWS de-identification pipeline for clinical notes. Partner data enters a hospital-owned AWS account with no internet access; SQS queues de-identification requests that launch tens or hundreds of Lambda jobs. Each job classifies note modality, assesses category-specific PHI risk, applies modality-specific preprocessing such as header/footer redaction and table extraction, runs John Snow Labs Healthcare NLP to detect identifiers, transforms detected PHI by redaction, hashing, or replacement, and then applies Hiding in Plain Sight (HIPS) postprocessing to lower re-identification risk. A QA/QC process with at least two clinically trained reviewers measures recall and precision, adjudicates discrepancies, retrains reviewers or revises processing when quality thresholds are missed, and only then moves approved output to the de-identified data store.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Partner hospital AWS account"] --> B["SQS de-identification queue"]
  B --> C["Lambda de-identification jobs"]
  C --> D["Modality risk classifier"]
  D --> E["Modality-specific preprocessor"]
  E --> F["John Snow Labs Healthcare NLP"]
  F --> G["PHI transformation engine"]
  G --> H["HIPS postprocessor"]
  H --> I["QA/QC reviewer panel"]
  I --> J["Hospital partner report"]
  I --> K["De-identified data store"]

Components

Data flow

For each hospital dataset, de-identification requests are placed on SQS, which launches Lambda jobs inside an internet-free AWS account. The jobs categorize each note by modality, reduce category-specific risk with preprocessing, run Healthcare NLP to flag PHI, scrub detected identifiers, apply HIPS replacement or date jitter, and send transformed output to human QA/QC. If reviewers confirm the results and thresholds are met, Dandelion shares a final report with hospital partners and moves the transformed dataset to a de-identified store; otherwise the transformed data is discarded and the raw data is reprocessed.

Why this architecture

Dandelion chose a risk-segmented pipeline because generic NLP over raw notes lacks enough context for messy clinical documents. Classifying modality first lets the system apply different preprocessing to low-risk radiology, high-risk interviews, obstetrics age phrases, scanned PDF footers, or ASCII tables. The air-gapped AWS design protects raw PHI, HIPS mitigates the impact of missed identifiers, and human expert review supplies the measured risk evidence required for HIPAA expert determination.

Models used: John Snow Labs Healthcare NLP · Healthcare NLP · HIPS dictionary replacement

Scale

• Dandelion describes clinical data covering millions of patients.
  “with our comprehensive longitudinal multimodal clinical data covering millions of patients we are building the largest AI ready training and validation data set in the world”transcript, stated
• The HIPAA privacy framing includes 18 identifiers.
  “this information is defined as the 18 Hippa identifiers seen here and can be used to identify contact or locate a single person”transcript, stated
• A dataset can fan out into tens or hundreds of Lambda de-identification jobs.
  “this in turn uh fires up tens or hundreds of deidentification Lambda jobs these jobs perform Transformations on any Phi”transcript, stated
• The HIPS approach is presented as reducing re-identification risk by about 90%.
  “based on Research over the past 10 years this approach is considered to reduce de identific re identification Risk by about 90%”transcript, stated
• Combining 95% PHI detection recall with HIPS is described as producing effective recall above 99%.
  “so if Phi detection recall is originally 95% with hips the effective recall is upwards of 99%”transcript, stated

Alternatives considered (roads not taken)

Run Healthcare NLP directly over every note and scrub all detected PHI

Rejected because: The speaker says the perfect-world approach fails because patient-note context is often malformed, extraneous, or embedded in document artifacts that confuse PHI detection.

Use HIPAA safe-harbor removal of specified identifiers

Rejected because: Dandelion chose expert determination to measure risk explicitly and permit more nuanced transformations.

Rely on raw redaction without HIPS replacement

Rejected because: Pure misses would make the missed token obvious, while HIPS mixes missed names and dates with plausible replacements so attackers cannot trivially identify which token is real.

Validate with a single human reviewer

Rejected because: The process requires at least two analysts per document and uses third-reviewer adjudication when reviewers disagree.

Patterns

privacy-preserving-processing (safety)

The pipeline isolates raw PHI in no-internet hospital AWS accounts, transforms identifiers through redaction, hashing, replacement, or date shifting, and applies HIPS so missed identifiers are less distinguishable from substituted tokens.

human-in-the-loop-gate (governance)

Clinically trained analysts manually review samples, at least two reviewers examine each document, discrepancies go to a third reviewer, and failed thresholds cause processing revision rather than release.

eval-driven-development (evaluation)

The de-identification process is driven by measured recall, precision cross-checking, reviewer discrepancy adjudication, and reruns when missed identifiers or false positives exceed quality thresholds.

tool-use-orchestration (architecture)

The system coordinates AWS SQS, Lambda workers, containerized air-gapped products, Healthcare NLP, preprocessing, HIPS postprocessing, and data-store routing as a controlled processing pipeline.

safety-policy-enforcement (governance)

HIPAA privacy requirements are translated into concrete release criteria: identifier categories, expert determination, de-identification transforms, recall reports, and partner-facing evidence before data egress.

Tradeoffs

Use expert determination instead of only removing specified identifiers

Chose expert determination with explicit risk measurement and nuanced transformations over safe-harbor removal of specified individual identifiers alone.

Rationale: The goal was to measure risk and preserve useful clinical information, not simply remove every specified field in the same way.

Segment free-text notes by modality before PHI detection

Chose modality categorization with risk-specific preprocessing over one generic Healthcare NLP pass over every note.

Rationale: Clinical note categories have different PHI risks and formatting patterns, so one generic pass misses important context and edge cases.

Apply HIPS replacement rather than making every miss obvious

Chose plausible token replacement with dictionary probabilities over pure redaction where missed names and dates stand out.

Rationale: HIPS cannot make detection perfect, but it lowers re-identification risk by hiding missed PHI among plausible substitutions.

Use double-review QA with adjudication

Chose two clinically trained analysts and third-reviewer triage for discrepancies over single-review validation or fully automated acceptance.

Rationale: Manual review quality itself must be controlled because disagreements may signal low reviewer quality or ambiguous review instructions.

Gotchas (what bit them)

Free-text clinical notes can destroy the context that PHI detection relies on.

Root cause: PDF conversion artifacts, footers, metadata, and line-by-line storage can place names or dates in positions where the NLP engine cannot infer what the token is.

Resolution: Dandelion added modality classification and category-specific preprocessing before Healthcare NLP, including header/footer redaction and table extraction.

ASCII tables can make identifier semantics invisible to normal NLP processing.

Root cause: When columns are represented by pipes and plus/minus delimiters, a date of birth can appear far from its column header and look like an ordinary date.

Resolution: The table structure must be extracted and processed independently before PHI classification.

Domain-specific codes can be confused with HIPAA identifiers.

Root cause: Five-digit CPT billing codes can resemble ZIP codes, and the correct interpretation depends on geography and surrounding clinical context.

Resolution: Use contextual checks such as hospital state, COVID-related meaning, and nearby address signals before deciding whether to treat a five-digit number as a ZIP code.

False positives can leak the HIPS masking mechanism.

Root cause: If a non-date value is mistakenly date-jittered, an attacker who knows the original value can infer the patient's date shift and reverse other dates.

Resolution: Dandelion performs precision cross-checking across HIPS dictionaries and date jitters, not recall-only validation.

Low-risk modality assumptions can still be wrong.

Root cause: Radiology or other low-risk categories may still include patient-detail headers, footers, scanned PDF artifacts, or other PHI-bearing details.

Resolution: Perform both manual and automated confirmation before treating a modality as low risk.

Evaluation

Dandelion evaluates de-identification with manual clinically trained reviewer samples, recall calculations for PHI redaction, precision cross-checks for HIPS and date-jitter false positives, reviewer discrepancy adjudication, and final partner reports broken down by HIPAA identifier category.

Metrics: PHI recall = true positive PHI redacted / (true positives + false negatives) · example 95% recall when five dates are missed out of 100 instances · HIPS estimated to reduce re-identification risk by about 90% · effective recall above 99% when 95% detection recall is combined with HIPS · at least two clinically trained reviewers per document sample · recall reported across individual HIPAA identifier categories

Outcomes

• Dandelion reports preserving valuable clinical data for clients while maintaining high security and patient privacy.
• The pipeline can reject transformed data and reprocess raw data when QA/QC does not confirm correct de-identification.
• HIPS is presented as improving effective recall from a 95% base PHI detection recall to above 99%.
• Hospital partners receive final reports with per-identifier recall counts to build confidence in the de-identification process.

The talk describes a production healthcare de-identification pipeline for longitudinal, multimodal clinical data covering millions of patients, running inside hospital AWS accounts with no internet access and scaling each dataset through tens or hundreds of Lambda jobs.