Building a Comprehensive LLM Platform for Healthcare Applications

includedhealth · Healthcare · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Healthcare GenAI platform

Included Health needed a reusable internal GenAI platform that could support healthcare LLM applications while managing provider routing, credentials, evaluation, prompting, serving, HIPAA-sensitive data concerns, and a rapidly changing model landscape.

Constraints:

healthcare use cases include HIPAA and regulated data concerns
multiple model providers need to be routed behind one internal interface
teams use different languages and need stable client interfaces
platform users need evaluation, prompt tooling, serving, and proxy access
the model/provider landscape changes too fast for a rigid long-term roadmap

Architecture

Included Health built Wordsmith around an internal proxy service, client library, serving infrastructure, evaluation library, prompt tooling, and internal ChatUI. All LLM requests pass through wordsmith-proxy, which centralizes provider routing and credentials. The platform supports applications such as clinical documentation, insurance plan questions, records collection, internal assistants, and emerging agent infrastructure.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  client-library[Client library] --> wordsmith-proxy[Wordsmith proxy]
  wordsmith-proxy --> model-providers[Model providers]
  wordsmith-proxy --> prompt-tooling[Prompt tooling]
  prompt-tooling --> evaluation-library[Evaluation library]
  wordsmith-serving[Wordsmith Serving] --> healthcare-applications[Healthcare applications]
  healthcare-applications --> agent-tooling[Agent tooling]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Client library`	Stable interface used by data scientists and engineers to call models.	`Wordsmith proxy`
`Wordsmith proxy`	Server-side routing and credential layer for LLM providers.	`Client library`, `Model providers`, `Prompt tooling`
`Model providers`	Azure OpenAI, Google VertexAI, AWS Bedrock, and other model backends.	`Wordsmith proxy`
`Evaluation library`	Wrapper around HuggingFace evaluate plus custom/open-source metrics.	`Prompt tooling`
`Prompt tooling`	Templates and few-shot example selection by semantic similarity.	`Wordsmith proxy`, `Evaluation library`
`Wordsmith Serving`	Model serving layer for capabilities such as Whisper transcription.	`Healthcare applications`
`Healthcare applications`	Clinical scribe, plan questions, records collection, internal ChatUI, and assistants.	`Wordsmith Serving`, `Agent tooling`
`Agent tooling`	wordsmith-tools and wordsmith-agents for tool-calling and agent configuration.	`Healthcare applications`

Data flow

A team calls Wordsmith through a client library or ChatUI. The request goes to wordsmith-proxy, which routes to the selected provider by model name, handles credentials centrally, and returns results. Supporting libraries provide evaluation metrics, prompt templates, few-shot example selection, transcription serving, and agent/tool configuration.

3 supporting quotes

“Instead of the request going from a Data Scientist’s local environment directly to Google, or OpenAI, all LLM requests at Included Health pass through a single internal service called wordsmith-proxy, which then routes them to the appropriate provider.”article, stated

“For our Data Scientists, this architecture means that they can switch between model providers without needing to write any new code, just by changing the model name from a string like “azure:gpt-4-turbo” to “google:gemini-pro”.”article, stated

“The wordsmith-evaluation library works as a wrapper around HuggingFace evaluate. You can use it with any of the existing metrics in the evaluate library, and we’ve also integrated some open-source metrics that aren’t in the library, as well as custom metrics that we’ve developed in house.”article, stated

Why this architecture

Centralizing routing on the server insulated users from provider churn, simplified credential management, and let teams change model providers by model string rather than code changes. Open-source/common interfaces kept Wordsmith adaptable as providers and APIs changed.

3 supporting quotes

“Proxying on the server side has several practical benefits: the first is that users are insulated from churn as we work on routing logic and added providers.”article, stated

“It also makes provider credential management easier: instead of trying to figure out how to securely distribute separate credentials to each user, we could set up a single OpenAI API key, a single Google service account credential, etc that belongs to the proxy service.”article, stated

“Nobody can have a 12-month roadmap for their GenAI platform right now, because nobody (except maybe Sam Altman) has any idea what’s going to happen in this space in 12 months.”article, stated

Models used: Azure GPT-4 Turbo · Google Gemini Pro · Google VertexAI · AWS Bedrock · Azure OpenAI · Whisper · Claude

Scale

Wordsmith rolled out ChatUI to 400 internal users.
“We’ve now rolled it out to 400 internal users! The “Assistants” feature has spawned a number of helpful productivity assistants, like Meeting Summarizer, which summarizes Google Meet transcripts, and IH Dejargonizer, a play on the popular [Dejargonizer GPT](https://simonwillison.”article, stated
An internal Confluence agent is used by over 80 engineers.
“an agent in ChatIH that can query for internal Confluence documentation, which is now used by over 80 of our engineers.”article, stated
Included Health moved from no GPT-4 release to managing LLMs from three cloud providers within six months.
“Fast forward 6 months, and all of a sudden we have LLMs from three different cloud providers (Google VertexAI, AWS Bedrock, and Azure OpenAI) to manage.”article, stated

Alternatives considered (roads not taken)

Client-side provider routing

Rejected because: The team moved routing and credentials server-side to insulate users from churn and simplify credential handling.

“I made the decision early on that the logic of routing between model providers would be handled on the server.”article, stated

Custom proprietary interfaces everywhere

Rejected because: The team favored open-source tools and common interfaces to integrate with external tooling.

“I have tried very hard to use open source and/or common interfaces wherever possible, which ultimately improves our ability to integrate Wordsmith with external tools.”article, stated

Patterns

`model-routing` (model-ops)

wordsmith-proxy routes model requests to providers behind a single internal service.

“wordsmith-proxy, which then routes them to the appropriate provider.”article, stated

`eval-driven-development` (evaluation)

The platform includes a wrapper around HuggingFace evaluate plus open-source and custom metrics.

“we’ve also integrated some open-source metrics that aren’t in the library, as well as custom metrics”article, stated

`prompt-versioning` (prompt-ops)

Prompt tooling abstracts templates and few-shot example selection for LLM applications.

“makes it easy for users to generate prompts from string templates”article, stated

`tool-use-orchestration` (agent-design)

Wordsmith is adding OpenAPI-backed tool endpoints and configurable LLM agents.

“wordsmith-tools, which is a simple way to add endpoints with an OpenAPI spec to be called by LLMs”article, stated

Tradeoffs

Provider routing location

Chose server-side routing through wordsmith-proxy over client-side provider calls.

Rationale: Server-side routing centralizes credentials and provider churn.

“Proxying on the server side has several practical benefits”article, stated

Platform interface strategy

Chose OpenAI-like/common APIs where possible over bespoke APIs for every capability.

Rationale: Common interfaces improve compatibility with existing tools and reduce future rewrites.

“open-source LLM server tools have popped up that use the OpenAI schema as a common interface”article, stated

Gotchas (what bit them)

The GenAI platform roadmap changes faster than yearly planning.

Root cause: External providers and internal company needs change rapidly.

Resolution: Keep the platform modular and use open-source/common interfaces.

“Development of Wordsmith is still very active, because this space is still rapidly changing.”article, stated

Credential distribution becomes fragile if each user calls providers directly.

Root cause: Each provider has separate credentials and access patterns.

Resolution: Centralize credentials in wordsmith-proxy.

“we could set up a single OpenAI API key, a single Google service account credential, etc that belongs to the proxy service.”article, stated

Evaluation

Wordsmith includes an evaluation library around HuggingFace evaluate, custom metrics, prompt tooling, and application-specific evaluation for deployed use cases.

Metrics: HuggingFace evaluate metrics · custom metrics · application adoption · internal users · engineer usage

“You can use it with any of the existing metrics in the evaluate library”article, stated

Outcomes

Wordsmith supports clinical scribe, insurance plan questions, records collection, data documentation, and internal ChatUI applications.
ChatUI rolled out to 400 internal users.
Agent tooling is now used by over 80 engineers for Confluence documentation queries.

The platform supports multiple production healthcare applications and 400 internal ChatUI users.

“It now covers both chats and calls, using transcription capabilities via a Whisper model hosted on Wordsmith Serving.”article, stated