AI-Powered Clinical Outcome Assessment Review Using Generative AI

clario · Healthcare · 2025 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Clinical outcome assessment interview review

Clario needed to standardize and scale review of multilingual clinical outcome assessment interviews while reducing expert-review variability and preserving regulatory and privacy requirements.

Constraints:

• COA interviews influence trial reliability and regulatory submissions
• manual audio-video review is time-consuming and variable
• reviews span multilingual data at global scale
• HIPAA and clinical-research compliance requirements apply
• AI summaries can sound plausible but be inaccurate

Architecture

Clario built a secure COA review pipeline: recordings are ingested, diarized, transcribed, embedded into OpenSearch, and reviewed by a graph-based agent that retrieves guides/checklists, validates factuality, and emits cited findings for human review.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  secure-recording-ingestion["Secure recording ingestion"]
  diarization-and-transcription["Diarization and transcription"]
  transcript-vectorization["Transcript vectorization"]
  guide-and-checklist-retrieval["Guide and checklist retrieval"]
  graph-based-agent-system["Graph-based agent system"]
  factual-validation-layer["Factual validation layer"]
  structured-findings["Structured findings"]
  human-clinical-review["Human clinical review"]
  secure-recording-ingestion --> diarization-and-transcription
  diarization-and-transcription --> transcript-vectorization
  transcript-vectorization --> graph-based-agent-system
  guide-and-checklist-retrieval --> graph-based-agent-system
  graph-based-agent-system --> factual-validation-layer
  factual-validation-layer --> structured-findings
  structured-findings --> human-clinical-review
  human-clinical-review --> secure-recording-ingestion

Components

Data flow

The system combines diarization, multilingual transcription, semantic search, and graph-based agent review over transcript segments, assessment guides, and checklist criteria.

Why this architecture

Factual validation is built in because early LLM summaries could sound plausible but be inaccurate in a regulated clinical-research workflow.

Models used: Whisper · Amazon Titan Text Embeddings v2 · large language models

Scale

• The target impact includes over 90% lower manual review effort and up to 100% data coverage compared with human-only spot checks.
  “Potential to decrease manual review effort by over 90%. * Quality improvements * Up to 100% data coverage through automated review versus human-only review of a smaller subset of recordings”article, stated
• The target is more than 90% reduction in manual review effort.
  “Potential to decrease manual review effort by over 90%. * Quality improvements * Up to 100% data coverage through automated review versus human-only review of a smaller subset of recordings”article, stated
• Automated review can expand quality checks to up to 100% data coverage.
  “Potential to decrease manual review effort by over 90%. * Quality improvements * Up to 100% data coverage through automated review versus human-only review of a smaller subset of recordings”article, stated
• Central review turnaround could shrink from weeks to hours.
  “Potential to shorten turn-around time by decreasing central review time from weeks to hours.”article, stated
• Clario cites more than 30,000 endpoint-data solution deployments and support for over 710 novel-drug regulatory approvals.
  “Since its founding over 50 years ago, Clario has deployed endpoint data solutions over 30,000 times, supporting over 710 novel drug regulatory approvals across more than 100 countries.”article, stated

Alternatives considered (roads not taken)

Human-only review of a smaller recording subset

Rejected because: Clario uses automated review to expand coverage while reserving expert attention for targeted interventions and remediation.

Plausible LLM summaries trusted without source cross-checking

Rejected because: Clario added factual validation after early LLM outputs sounded plausible but were inaccurate.

Manual expert review as the only quality-control path

Rejected because: Automated review expands coverage and lets experts focus on targeted interventions.

Patterns

tool-use-orchestration (orchestration)

The system coordinates diarization, transcription, embedding, retrieval, checklist loading, and graph-based agent review.

rag-retrieval (retrieval)

Transcribed interview segments are embedded into OpenSearch so the review agent can retrieve relevant evidence for each checklist criterion.

multi-agent-orchestration (orchestration)

The review engine uses a graph-based agent workflow with predefined decision nodes to perform structured COA checks.

safety-policy-enforcement (safety)

AI outputs are cross-checked against source documents before human review to reduce plausible but inaccurate summaries.

eval-driven-development (evaluation)

Clario regularly compares models and configurations on custom datasets to maintain reliability.

Tradeoffs

How much review coverage to automate.

Chose automated review across up to all recordings with targeted human interventions over human-only review of smaller sampled subsets.

Rationale: Automation can expand coverage and focus human reviewers on raters or sites needing remediation.

How to combine automation and responsible review.

Chose AI review with factual validation before human review over direct acceptance of generated interview summaries.

Rationale: The team found hallucination-like summaries during testing and added source cross-checking before review.

How much clinical-review evidence coverage to pursue

Chose automated review for broad coverage plus targeted human intervention over human-only spot checking of a smaller subset.

Rationale: The source frames up to 100% automated coverage as a way to improve quality while focusing experts where remediation is needed.

Gotchas (what bit them)

Manual expert review introduced variability into clinical-trial quality checks.

Root cause: Different reviewers naturally vary in how they judge interviews and assessment technique.

Resolution: Use standardized automated review against checklist criteria and source-backed findings.

LLMs generated plausible but inaccurate summaries during initial testing.

Root cause: Generated summaries can appear credible while diverging from source recordings or transcripts.

Resolution: Cross-check AI outputs against source documents before human review.

Clinical-research automation must satisfy security and compliance at scale.

Root cause: COA recordings are sensitive clinical-trial data subject to privacy and regulatory requirements.

Resolution: Use encrypted ingestion/storage and HIPAA-compliant AWS services with audit and governance support.

Evaluation

Clario evaluates the system with source cross-checking, continuous model evaluation, custom datasets, comparative studies across models/configurations, and clinical operational metrics.

Metrics: potential to decrease manual review effort by over 90% · up to 100% data coverage · potential to shorten central review from weeks to hours · over 30,000 endpoint data solution deployments · over 710 novel drug regulatory approvals supported

Outcomes

• The solution shows potential for over 90% manual-review effort reduction.
• Automated review can expand coverage beyond human sampled spot checks.
• Clario expects faster targeted remediation and stronger data reliability for submissions.

AWS customer article uses potential/initial implementation language; results should be treated as promising but not fully independent proof.