AI-Powered Sleep Coach for CBTI Protocol Delivery

rest · Healthcare · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Sleep coaching and wellness behavior change

Rest wanted to make CBTI-inspired sleep coaching more accessible for people struggling with insomnia-like sleep problems, while preserving trust, personalization, clinical boundaries, and daily adherence in a sensitive wellness domain.

Constraints:

• CBTI-trained practitioners can be expensive and have long waitlists
• users must log sleep daily and change difficult routines over an eight-week program
• the product must not diagnose, treat, or give medication advice
• voice and text stacks had to stay behaviorally aligned despite different platforms
• personalization required combining structured sleep logs, unstructured conversations, memory, and CBTI knowledge

Architecture

Rest evolved from a text chatbot into a voice-first AI sleep coach. Users log sleep every morning, then receive a short report and a personalized check-in. The coach uses OpenAI models for reasoning, Vapi for voice agent infrastructure, file-based RAG for sleep/CBTI knowledge, a custom memory process that extracts user facts from past conversations, and dynamic agenda generation that creates prioritized prompt snippets based on last night's sleep, program stage, recent data, and relevant memories. Weekly error analysis, Langfuse traces, domain-expert review, LLM-powered safety evals, and guardrails around medication/medical advice guide prompt, RAG, and memory improvements.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  sleep-log-flow["sleep-log-flow"] --> dynamic-agenda-generator["dynamic-agenda-generator"]
  dynamic-agenda-generator --> coach-checkin-interface["coach-checkin-interface"]
  coach-checkin-interface --> openai-text-agent["openai-text-agent"]
  coach-checkin-interface --> vapi-voice-agent["vapi-voice-agent"]
  openai-text-agent --> file-based-rag-store["file-based-rag-store"]
  vapi-voice-agent --> file-based-rag-store
  openai-text-agent --> user-memory-store["user-memory-store"]
  vapi-voice-agent --> user-memory-store
  post-conversation-memory-extractor["post-conversation-memory-extractor"] --> user-memory-store
  user-memory-store --> dynamic-agenda-generator
  openai-text-agent --> langfuse-trace-and-eval-layer["langfuse-trace-and-eval-layer"]
  vapi-voice-agent --> langfuse-trace-and-eval-layer
  langfuse-trace-and-eval-layer --> weekly-error-analysis["weekly-error-analysis"]

Components

Data flow

A user logs numeric sleep diary data after waking. Rest generates a short report, then prepares a tailored agenda before the check-in. The agenda process gathers last-night sleep data, previous-week context, program stage, and memories extracted from earlier conversations. In text mode, the agent runs through OpenAI's assistant stack; in voice mode, Vapi handles speech-to-text, streaming, and text-to-speech while still using OpenAI as the LLM provider. The agent can retrieve sleep/CBTI knowledge from file-based RAG, run the agenda-driven conversation, and after the conversation another prompt extracts memories for future sessions. Traces feed weekly annotation/error analysis and safety evals.

Why this architecture

Rest chose voice because trust, intimacy, and fast disclosure matter when users discuss sleep struggles and must change habits. It kept the initial architecture simple, using OpenAI Assistants and later Vapi rather than a fully custom LangGraph-style agent, because the team wanted to learn from users before over-engineering. RAG moved general CBTI knowledge out of an overgrown system prompt, memory provided continuity across the eight-week journey, and dynamic agendas let the coach start each day with program-aware goals rather than waiting for the user to ask.

Models used: OpenAI GPT-4 · OpenAI Assistants · Vapi · Langfuse · Hamming · file-based RAG

Scale

• Rest's protocol is framed as an eight-week sleep program.
  “We help people during eight weeks to solve their sleep problems”transcript, stated
• The founders cite CBTI efficacy for 80% of people with lasting effects for years.
  “efficacy for 80% of people and efficacy that lasts for years”transcript, stated
• Traditional CBTI can cost thousands of dollars out of pocket.
  “need to pay out of pocket thousands of dollars.”transcript, stated
• The scientific advisor example involved a year-and-a-half waitlist.
  “Dr. RJ Mason, she's at UCSF and that's exactly her case. Like she has a year and a half weight list.”transcript, stated
• Rest saw roughly 10% of podcast-app users using it for sleep.
  “It was probably more like 10% something like that.”transcript, stated
• Voice became the majority interaction mode and logs hundreds of conversation hours monthly.
  “it's the majority of the user interactions and we're logging like hundreds of hours of voice conversations every month”transcript, stated
• Dynamic agendas generate three prioritized topics before daily check-ins.
  “come up with three items that we should be discussing in order of priority”transcript, stated
• Rest performs weekly error analysis and weekly user interviews for prioritization.
  “weekly error analysis process.”transcript, stated

  “for the most part every week we interview a user”transcript, stated

Alternatives considered (roads not taken)

sleep audio content as the main product

Rejected because: Audio helped some users fall asleep, but the team saw a deeper problem around chronic sleep habits and CBTI-style protocol guidance.

text-only chatbot

Rejected because: Users opened up more and had longer interactions with voice.

one huge system prompt

Rejected because: The context became too large and made it harder for the LLM to focus on the right knowledge at the right moment.

fully custom LangGraph or memory architecture upfront

Rejected because: The team judged it premature before learning the exact problems from users and error analysis.

Patterns

rag-retrieval (knowledge-grounding)

General sleep, CBTI, and expert-codified knowledge moved from the system prompt into file-based retrieval used by both text and voice agents.

agent-memory (context-management)

A post-conversation prompt extracts user memories, and later conversations retrieve recent or relevant memories to personalize coaching.

eval-driven-development (quality)

Weekly error analysis, domain-expert review, trace annotation, and LLM evals determine AI engineering priorities.

guardrails-io-filtering (safety)

The coach has explicit boundaries against diagnosis, medication discussion, and medical advice, with LLM evals to catch violations.

prompt-versioning (model-operations)

Rest tracks prompt versions across the text and voice branches, though the team notes branch drift is messy.

Tradeoffs

Use third-party managed agent/voice platforms before building a custom agent stack.

Chose OpenAI Assistants plus Vapi over full custom LangGraph and memory architecture upfront.

Rationale: Managed platforms reduced initial engineering burden and let the team focus on user value, but they also reduced control.

Keep user data central in the prompt while moving general knowledge to RAG.

Chose always-present user context plus retrieved sleep/CBTI knowledge over putting all user information behind semantic retrieval.

Rationale: Personalization needed to be consistently available, while general knowledge could be retrieved when relevant.

Use voice despite privacy/context concerns.

Chose voice-first coaching for most interactions over text-only chat.

Rationale: Voice increased intimacy, disclosure, and interaction length, but required maintaining a second technical branch.

Gotchas (what bit them)

Wellness coaching can drift toward medical advice.

Root cause: The product discusses sleep, CBTI principles, and user struggles in a health-adjacent context.

Resolution: Avoid diagnosis, medication advice, and therapy claims; use disclaimers, legal review, experts, and safety evals.

Maintaining text and voice parity became difficult.

Root cause: Text and voice use different agent stacks and do not always interpret prompt/tool changes the same way.

Resolution: Version prompts and adapt changes per branch, while recognizing the voice branch currently leads.

Memory can become stale or overlong.

Root cause: User context like travel or temporary sleep disruptions may matter briefly but not forever.

Resolution: Use recency heuristics, explore expiration dates, and split recent versus older memories.

RAG tool use did not work automatically after moving knowledge out of the prompt.

Root cause: The agent needed immediate-context cues and prompt guidance to know when retrieval was appropriate.

Resolution: Add explicit prompt guidelines about the moments where retrieval should happen.

Evaluation

Rest evaluates quality through weekly error analysis, domain-expert review of conversations, Langfuse annotation queues, LLM-powered safety/boundary evals, user interviews, customer support signals, and experiments with voice-scenario replication in Hamming.

Metrics: weekly error analysis · weekly user interviews · hundreds of hours of voice conversations per month · LLM-powered medication/medical-advice safety evals · manual annotation queues from everyday conversation traces

Outcomes

• Rest moved from text chat to voice as the majority interaction mode
• users log sleep daily and receive agenda-driven coaching check-ins
• RAG, memory, and dynamic agenda generation were added in response to observed failures
• the product now logs hundreds of hours of voice conversations monthly
• the team maintains explicit wellness/medical boundaries and safety evaluations

The source reports an eight-week protocol, approximately 10% original podcast sleep use, hundreds of voice hours/month, weekly error analysis, and weekly user interviews.