AI-Powered Healthcare: Building Reliable Care Agents in Production

sword_health · Healthcare · 2025 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Healthcare AI for remote physical therapy

Sword Health needed to ship LLM-powered patient and therapist support in a regulated healthcare setting where Phoenix could help patients during and outside sessions while preserving clinical safety, reliability, and compliance.

Constraints:

• healthcare outputs must be safe, consistent, and reliable
• clinical decisions and feedback require human review
• real-time patient interactions make guardrail latency visible
• domain knowledge must stay current and clinically bounded
• developers and reviewers need safe access to production-like data for evaluation

Architecture

Sword Health built Phoenix as an AI care agent around a layered safety and learning loop. Patient and therapist interactions pass through task-specific input/output guardrails, with medical-advice guardrails constrained by clinician-authored guidelines. For domain questions, the system retrieves knowledge articles from a vector database and places them in the prompt. Development is governed by evaluation loops: offline evals, live checks, subject-matter-expert review in an internal Gondola tool, LLM-as-judge alignment, A/B tests, product metrics, manual audits, and repeated production-data review. Clinical recommendations remain human-gated by physical therapists.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  phoenix-care-agent["phoenix-care-agent"] --> input-guardrails["input-guardrails"]
  input-guardrails --> phoenix-care-agent
  phoenix-care-agent --> rag-knowledge-retriever["rag-knowledge-retriever"]
  rag-knowledge-retriever --> vector-database["vector-database"]
  rag-knowledge-retriever --> knowledge-article-store["knowledge-article-store"]
  phoenix-care-agent --> output-guardrails["output-guardrails"]
  output-guardrails --> clinical-guidelines["clinical-guidelines"]
  phoenix-care-agent --> physical-therapist-back-office["physical-therapist-back-office"]
  physical-therapist-back-office --> gondola-human-review["gondola-human-review"]
  evaluation-suite["evaluation-suite"] --> gondola-human-review
  evaluation-suite --> llm-as-judge-evaluator["llm-as-judge-evaluator"]
  monitoring-and-audits["monitoring-and-audits"] --> evaluation-suite
  phoenix-care-agent --> memory-system["memory-system"]
  memory-system --> vector-database

Components

Data flow

A patient interacts with Phoenix during or outside a rehab session. Input guardrails filter unsafe or adversarial content before the model; RAG retrieves relevant knowledge articles when domain knowledge is needed; the model generates a response or recommendation; output and medical-advice guardrails constrain what can reach the patient. For clinical recommendations, Phoenix acts as a therapist co-pilot: the physical therapist reviews, accepts, rejects, or changes the suggestion in the back office. Those reviewed outputs, user feedback, manual audits, and eval results feed future guardrails, prompts, datasets, and monitoring.

Why this architecture

The architecture separates generation, clinical control, evaluation, and data review because healthcare failures are high-impact and LLM outputs are non-deterministic. Guardrails bound unsafe input/output, RAG supplies domain knowledge without fine-tuning, human review protects clinical decisions, and repeated eval/feedback loops keep the system improving after release. Sword also keeps memory and production-data review close to the product because Phoenix needs patient context over time and new failure modes appear only in real use.

Models used: large language models · Claude 3.5 Sonnet · GPT-4.0

Scale

• Sword Health operates three remote-physical-therapy product lines: Thrive, Move, and Bloom.
  “We have three main products: Thrive, Move, and Bloom. Thrive covers chronic pain, Move, pain prevention, and Bloom, pelvic healthcare.”article, stated
• Online guardrails increased Phoenix latency by about 30%.
  “When adding online guardrails for Phoenix, we observed an increase in latency of around 30%.”article, stated
• Switching from GPT-4.0 to Claude 3.5 Sonnet improved one task by about 10 percentage points.
  “just by switching from GPT-4.0 to Claude 3.5 Sonnet, and keeping everything mostly the same, just a few tweaks on prompt engineering, we observed an increase in performance of around 10% points”article, stated
• Conversation sentiment analysis showed around 50% non-engagement in one feedback view.
  “around 50% of the times, the patient does not engage us in conversation.”article, stated
• Human and model evaluations can use a 1-to-5 score or binary judgments depending on task.
  “You can ask humans to evaluate the outputs in a score from, for example, 1 to 5, or binary, like I described.”article, stated

Alternatives considered (roads not taken)

Let Phoenix provide clinical feedback directly to patients

Rejected because: Sword keeps clinical recommendations under physical-therapist review for legal and safety reasons.

Use only online guardrails in real-time conversations

Rejected because: Real-time guardrails added latency, so Sword optimized online checks and used offline post-conversation analysis where streaming remains hard.

Use the model's long context window instead of retrieval

Rejected because: Sword observed lost-in-the-middle behavior and treats RAG as the better path for adding domain knowledge.

Jump straight to fine-tuning for every quality problem

Rejected because: Sword starts with prompt engineering and uses RAG when the model needs knowledge; fine-tuning is for behavior/tone/instruction-following gaps.

Rely on a single evaluation style

Rejected because: Human, non-LLM, and LLM-based ratings cover different tasks and failure modes.

Patterns

guardrails-io-filtering (governance)

Phoenix uses input guardrails for prompt injection/jailbreak/content safety and output guardrails for content safety, structure, and medical-advice boundaries.

human-in-the-loop-gate (governance)

Clinical recommendations are held behind physical-therapist review before they affect the patient.

rag-retrieval (retrieval)

Sword embeds support articles, retrieves the top similar articles, and includes them in the prompt for patient-support answers.

eval-driven-development (evaluation)

Prompt/model iterations move through offline evaluations, live checking, human expert review, A/B testing, and monitoring before promotion.

llm-as-judge (evaluation)

Sword uses LLM-based rating as a middle ground between human review and deterministic metrics, with alignment checks against humans.

agent-memory (context)

Phoenix stores current-user history, past healthcare data, peer decisions, and learned context across MySQL and vector databases.

data-quality-loop (data-pipeline)

Sword repeatedly reviews production inputs/outputs to discover new failure modes, build datasets, and improve guardrails/evals/prompts.

Tradeoffs

Keep clinical recommendations under therapist control.

Chose Phoenix as a physical-therapist co-pilot with accept/reject/change review over autonomous clinical feedback delivered directly to patients.

Rationale: The human gate preserves legal accountability and clinical safety while still allowing Phoenix to draft recommendations.

Use task-specific guardrails instead of one generic safety threshold.

Chose product/context-specific guardrail thresholds and medical-advice rules over generic safety filters applied uniformly across healthcare products.

Rationale: Generic filters can create false positives in legitimate pelvic-health language, while medical advice needs narrower clinical constraints.

Choose optimization path by failure mode.

Chose prompt engineering first, RAG for missing knowledge, fine-tuning for behavior over a single linear prompt-to-RAG-to-fine-tune progression.

Rationale: Different methods solve different problems: prompt engineering establishes the target, RAG adds knowledge, and model optimization changes behavior/tone.

Use RAG rather than dumping knowledge into long context.

Chose article-level retrieval from a vector database over relying on the model's large context window.

Rationale: Sword observed lost-in-the-middle behavior and treats retrieval quality as separately measurable.

Mix evaluation approaches instead of standardizing on one evaluator.

Chose human, non-LLM, and LLM-based ratings over only human review or only automated evaluation.

Rationale: Human review captures nuance but is slow; deterministic metrics scale but miss subjectivity; LLM judges sit between them and must be aligned to humans.

Gotchas (what bit them)

Guardrails can add too much latency for real-time patient feedback.

Root cause: Online safety checks sit in the response path for time-sensitive Phoenix conversations.

Resolution: Optimize online guardrails and move some checks into offline post-conversation analysis when streaming makes online checks impractical.

Guardrails can block valid healthcare content through false positives.

Root cause: Generic content-safety thresholds misread domain-specific language, especially in pelvic health.

Resolution: Make guardrails task-specific and tune thresholds per product context.

LLM releases regress unpredictably without evaluation suites.

Root cause: Large language models are non-deterministic, so prompt/model changes do not behave like deterministic software changes.

Resolution: Treat evals as LLM unit tests and run offline checks, expert review, A/B tests, and ongoing monitoring before/after release.

RAG can fail because retrieval misses or pollutes the needed context.

Root cause: Generation quality depends on retrieval relevance, context precision/recall, and whether the answer exists in the knowledge base.

Resolution: Evaluate generation and retrieval separately with RAGAS-style metrics and use query rewriting or clarification when similarity is low.

Sensitive healthcare data can make hands-on debugging and inspection hard.

Root cause: Regulated patient data raises HIPAA/GDPR-style access and anonymization concerns for developers.

Resolution: Use legal allowances for care improvement, explore anonymization, and promote safe data-review tooling and culture.

Evaluation

Sword evaluates LLM features through test sets, offline evaluations, live checks, subject-matter-expert review, LLM-as-judge alignment, A/B tests, product metrics, manual audits, and repeated production-data review.

Metrics: human 1-to-5 or binary ratings · BLEU · ROUGE · Sequence Matcher score · LLM-human agreement · RAGAS faithfulness · RAGAS relevance · context precision · context recall · acceptance metrics · product metrics · manual audit findings

Outcomes

• Sword shipped multiple LLM-powered features supporting patients during sessions, patients outside sessions, and physical therapists.
• Phoenix provides real-time patient feedback and non-clinical support while remaining a therapist co-pilot for clinical recommendations.
• The team learned concrete production lessons around guardrail latency, task-specific thresholds, eval-driven release, RAG evaluation, feedback collection, and repeated data inspection.

The source reports three product lines, a 30% latency hit from online guardrails, a 10 percentage-point model-switch gain in one example, and memory built from current-user data, past healthcare data, peer decisions, MySQL databases, and vector databases.