Healthcare Patient Journey Analysis Platform with Multimodal LLMs

john_snow_labs · Healthcare · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: healthcare multimodal patient-journey analytics

John Snow Labs needed an enterprise healthcare platform that could unify messy structured, unstructured, FHIR, and imaging-derived patient data into longitudinal patient journeys, then answer natural-language clinical and cohort questions with references, explanations, and production-grade privacy controls.

Constraints:

• patient histories span years of visits, claims, prescriptions, labs, radiology, pathology, and external events
• real clinical data is dirty, inconsistent, duplicated, uncertain, and spread across modalities
• long-context LLMs are not practical for millions of patients and billions of documents
• healthcare users need answers with references and explanations, not just generated summaries
• the system must run within customer infrastructure and avoid sending PHI to external LLM APIs
• natural-language cohort queries require clinical terminologies, temporal reasoning, optimized database access, and business-logic explainability
• non-technical clinicians, nurses, administrators, and researchers must be able to use the platform without knowing schemas, SQL, or code sets

Architecture

John Snow Labs built a customer-hosted Kubernetes platform that ingests structured EHR data, unstructured clinical notes, FHIR resources, imaging-derived data, and other documents. Healthcare-specific LLMs extract clinical entities, assertion status, relations, codes, and dates; terminology services normalize those entities to SNOMED and other code systems; temporal normalization maps relative dates to concrete dates; and a reasoning/deduplication layer builds a patient knowledge graph with provenance, confidence, and traceability. The normalized record is stored in an OMOP-based relational database, extended where needed for explainability and confidence metadata. On top of that database, a natural-language query agent uses a RAG-like query-template tool, terminology expansion, schema/SQL adjustment, optimized execution, and explanation rendering so users can ask patient, cohort, risk-score, and decision-support questions without manually writing SQL.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["Customer infrastructure deployment"] --> B["Multimodal data connectors"]
  A --> C["Kubernetes runtime"]
  B --> D["Information extraction LLMs"]
  D --> E["Terminology service"]
  D --> F["Temporal normalization service"]
  E --> G["Reasoning and deduplication layer"]
  F --> G
  G --> H["OMOP relational store"]
  H --> I["Natural-language query agent"]
  I --> J["RAG query-template tool"]
  J --> K["Schema adjustment tool"]
  I --> E
  K --> L["Optimized query execution layer"]
  L --> H
  L --> M["Explanation and reference UI"]
  H --> N["BI and data science tools"]
  H --> O["Risk-score containers"]

Components

Data flow

Raw patient data enters through APIs, EHR stores, FHIR messages, PDFs, images, or structured databases inside the customer's infrastructure. Information-extraction models read free text and embedded text to extract clinical entities, assertion status, relations, clinical codes, and dates. Terminology services map source codes to standard vocabularies, temporal normalization resolves relative dates, and healthcare-specific deduplication merges repeated or conflicting events into a patient knowledge graph. The graph and traceability metadata are persisted to an extended OMOP relational database. A user then asks a natural-language question; the agent retrieves an efficient prebuilt query pattern, expands terms through the terminology service, adjusts schema and SQL for the requested dimensions, executes optimized queries, and returns answers with references and business-logic explanations.

Why this architecture

The platform rejects both clean-table assumptions and pure text-to-SQL because healthcare questions require messy multimodal ingestion, clinical normalization, temporal reasoning, optimized database execution, and explainable business logic. OMOP gives customers an open relational substrate that existing teams and tools understand, while the agent layer hides terminology, schema, and query complexity from clinicians and administrators. Keeping the system inside the customer's infrastructure is necessary because the data contains PHI.

Models used: John Snow Labs medical LLMs · Healthcare-specific LLMs · Healthcare NLP · terminology resolvers · RAG query-template component · GPT-4o comparison baseline

Scale

• The platform targets longitudinal patient histories spanning five to ten years.
  “looking at that entire patient history right so over you know five years 10 years the patient can have multiple hospital visits Radiology tests”transcript, stated
• The intended scale is millions of patients and billions of documents.
  “we're going to deal here with you know millions of patients billions of documents and we need to be able to update the data in an ongoing basis”transcript, stated
• Customer organizations can serve 10 million, 20 million, or 30 million patients per year.
  “we're working with organizations that serve 10 million patients 20 million patients 30 million patients per year”transcript, stated
• A typical cancer patient is described as generating over a thousand pages of text per year.
  “a typical cancer patient generates over a thousand pages of text per year in the US everything from you know chemo Radiology psychological screenings”transcript, stated
• The extraction library covers more than 400 entity types.
  “right now with Johnson Labs there are more than 400 types of entities we know to automatically extract from free text”transcript, stated
• Structured clinical codes may miss one-third to one-half of relevant patients for some cohort questions.
  “the right codes are going to be missing and not from 5% of the patients they're going to be missing usually from between like one3 to one half of the patients”transcript, stated
• The implementation project is described as a 12-week engagement.
  “we only do this now with a 12we implementation project in the Implement implementation project we partner with your team”transcript, stated

Alternatives considered (roads not taken)

Analyze clean relational tables or single documents only

Rejected because: The platform is built for raw dirty clinical data, unstructured notes, inconsistent codes, conflicting labs, and multimodal patient histories rather than clean analytic tables.

Put all patient context into a long-context LLM

Rejected because: The speaker says this is meaningless at the intended corpus scale of millions of patients, billions of pages, and very large longitudinal histories.

Build a proprietary clinical data model from scratch

Rejected because: John Snow Labs chose OMOP because an open relational standard lets multiple groups and tools interact with the data and extend it where needed.

Treat natural-language cohorting as ordinary text-to-SQL

Rejected because: Healthcare queries require terminology expansion, temporal logic, optimized query templates, schema adjustments, and explanations that a single text-to-SQL generation step cannot reliably provide.

Let generated SQL run freely over the database

Rejected because: The speaker warns that unconstrained query generation can scan hundreds of millions of rows and crash the server, so the agent must use optimized query patterns, indexes, materialized views, and caching.

Use general-purpose LLMs for clinical tasks

Rejected because: John Snow Labs argues for healthcare-specific LLMs, citing blind randomized physician evaluation where its tuned medical models beat GPT-4o on clinical summarization, extraction, and question answering.

Patterns

tool-use-orchestration (agent architecture)

The cohort-query layer is an explicit multi-step agent that calls a query-template retrieval tool, terminology service, schema-adjustment tool, optimized execution path, aggregation, and explanation renderer instead of emitting a one-shot SQL query.

rag-retrieval (retrieval)

The agent retrieves similar prebuilt queries that are already known to run efficiently and accurately, then adapts them to the user's specific request.

fine-tuning-adaptation (model adaptation)

John Snow Labs uses its own healthcare-specific LLMs trained and fine-tuned on real-world medical data for extraction, summarization, query understanding, and medical question answering.

eval-driven-development (evaluation)

The medical models were benchmarked through blind randomized evaluations by practicing physicians across factuality, clinical relevance, and conciseness before being positioned as better than GPT-4o for the platform tasks.

structured-output-contract (data contract)

The platform normalizes extracted multimodal facts into an extended OMOP relational schema so downstream SQL, dashboards, notebooks, and natural-language queries operate over a stable open data model.

privacy-preserving-processing (safety)

The platform is deployed inside the customer's infrastructure, runs John Snow Labs' own LLMs locally, and avoids sending PHI to John Snow Labs, OpenAI, cloud providers, or other external APIs.

observability-tracing (traceability)

The extracted patient model carries source references, traceability, confidence levels, and explanation metadata so users can inspect where answers and cohort membership came from.

safety-policy-enforcement (governance)

Healthcare-specific business logic, query explanations, and PHI deployment boundaries enforce clinical and privacy requirements rather than letting users run opaque or unsafe generated SQL.

Tradeoffs

Use an open OMOP relational model with extensions

Chose an open standard relational model extended for traceability and confidence over a proprietary model that perfectly matched internal requirements.

Rationale: OMOP gives healthcare teams, BI tools, SQL users, data scientists, and open-source ecosystem tools a common substrate, while targeted extensions cover missing traceability fields.

Build an agentic query workflow instead of one-shot text-to-SQL

Chose a multi-step agent with query retrieval, terminology expansion, schema adjustment, execution, aggregation, and explanation over direct natural-language-to-SQL generation.

Rationale: Healthcare cohort questions require complex reasoning over terminology, schemas, time windows, and optimized database operations; the speaker says text-to-SQL alone is insufficient.

Constrain query execution to optimized patterns

Chose prebuilt efficient query types, indexes, materialized views, and caching over allowing arbitrary SQL generated from user prompts.

Rationale: The system targets end users who do not understand database performance, so the agent must avoid unsafe full-table scans over huge clinical tables.

Run everything inside customer infrastructure

Chose customer-hosted software and models with no external LLM API calls over sending PHI to John Snow Labs, OpenAI, cloud LLM APIs, or third-party services.

Rationale: The platform handles PHI and is sold into healthcare organizations that require data, processing, and custom IP to remain inside their environment.

Use healthcare-specific LLMs rather than general-purpose models

Chose John Snow Labs' medical LLMs trained and evaluated on clinical tasks over general-purpose GPT-4o-style models for extraction, summarization, and query answering.

Rationale: The source argues that clinical accuracy, consistency, and domain terminology require specialized models, and reports physician blind evaluations favoring the specialized models.

Return explanations and references instead of opaque answers

Chose answers with source references, highlights, business logic, and patient-level provenance over black-box natural-language answers.

Rationale: Healthcare users need to verify why a patient matched a cohort or why a conclusion was made, especially when definitions differ by role and use case.

Gotchas (what bit them)

Clean-table assumptions fail on real patient journeys.

Root cause: Patient data arrives dirty, inconsistent, incomplete, uncertain, duplicated, and spread across structured records, notes, FHIR, claims, outside providers, and imaging or document systems.

Resolution: The platform starts from raw multimodal data, extracts and normalizes facts, and builds a unified patient journey rather than assuming a clean database is already available.

Long-context LLMs do not solve patient-scale longitudinal analytics.

Root cause: The target corpus is not a single document or million-character prompt; it spans millions of patients, billions of pages, and continuously updated records.

Resolution: John Snow Labs normalizes extracted facts into an OMOP relational store and uses optimized queries rather than putting all patient context into the model prompt.

Clinical codes alone miss too much patient evidence.

Root cause: Diagnoses, medications, comorbidities, visits, and other history can be missing from structured codes even when they appear in notes or external sources.

Resolution: The system combines structured, unstructured, FHIR, and other modalities so downstream coding, risk adjustment, decision support, and analytics see richer evidence.

Generated healthcare SQL can be both wrong and operationally dangerous.

Root cause: Healthcare cohort questions require complex joins, large tables, terminology hierarchies, and time windows; unconstrained generated SQL may produce inaccurate answers or heavy scans.

Resolution: The agent retrieves known efficient query patterns, expands terms, adjusts schema, and uses optimized indexes, materialized views, and caching.

The same clinical term means different things to different users.

Root cause: Billing, clinical, quality, research, and departmental users may define diabetes, patient membership, gender, or cohort inclusion differently.

Resolution: The system must expose business logic and patient-level provenance so users can inspect, challenge, and adjust how a query was interpreted.

Deduplicating longitudinal clinical facts is field-specific rather than generic.

Root cause: Repeated mentions, varying specificity, changing symptoms, and multiple measurements need different merge logic depending on the clinical field.

Resolution: The reasoning layer deduplicates and summarizes with healthcare-specific logic, preserving uncertainty and choosing the most relevant specificity or aggregate.

Evaluation

John Snow Labs evaluated its medical LLMs through blind randomized physician comparisons against GPT-4o on clinical text summarization, clinical information extraction, and medical question answering, asking doctors to judge factuality, clinical relevance, and conciseness. The platform also emphasizes consistency as a production requirement for user trust, optimized query execution as an operational guard, and explanation/provenance as a user-facing correctness check.

Metrics: blind randomized practicing-physician evaluation · three evaluated tasks: clinical text summarization, clinical information extraction, and medical question answering · three physician-rated dimensions: factuality, clinical relevance, and conciseness · John Snow Labs reports nearly 2:1 preference over GPT-4o on those dimensions · consistency requirement: same question should return the same answer repeatedly · query safety requirement: avoid broad scans over 200-300 million row tables

Outcomes

• The system is described as ready to deploy today through a 12-week implementation project.
• John Snow Labs reports deployments or deployability across on-premise environments, AWS, Azure, Databricks, Snowflake, and Oracle.
• The platform broadens access from BI/data specialist teams to clinicians, nurses, administrators, and researchers asking natural-language questions.
• The system can productize customer risk models so they update for every patient when data changes and carry provenance, versioning, and confidence.
• The speaker claims richer multimodal data improves clinical decision support, coding, revenue cycle, risk adjustment, population health, and downstream analytics.

The platform is positioned for enterprise healthcare organizations serving 10-30 million patients per year, with millions of patients, billions of documents, 200-300 million row tables, and customer-specific 12-week implementation projects.