Building an AI Teaching Assistant: ChatLTV at Harvard Business School

harvard · Education · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: MBA course teaching assistant

Harvard Business School needed a course-specific AI faculty co-pilot that could answer student questions from thirteen years of Launching Tech Ventures course material while protecting copyrighted content and giving faculty visibility into student confusion.

Constraints:

The corpus included copyrighted HBS content that could not flow into public model training.
Students used Slack as part of the course workflow.
The course corpus was large and heterogeneous: cases, notes, books, decks, spreadsheets, Slack Q&A, glossaries, and interviews.
The system needed both student utility and faculty observability.

Architecture

ChatLTV is a Slack-based RAG chatbot over the LTV course corpus. It retrieves relevant chunks from Pinecone, passes the student question and context to GPT-4 through Azure OpenAI for privacy and compliance, uses LangChain middleware, returns answers and sources in Slack, and includes a CMS for faculty to add content, observe queries, and add administrative course material.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  student[Student] --> slack-app[Slack App]
  slack-app --> langchain-middleware[LangChain Middleware]
  langchain-middleware --> pinecone-vector-db[Pinecone Vector DB]
  pinecone-vector-db --> course-corpus[Course Corpus]
  langchain-middleware --> azure-openai-gpt-4[Azure OpenAI GPT-4]
  cms[CMS] --> course-corpus[Course Corpus]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Student`	MBA student asking course questions privately or publicly.	`Slack App`
`Slack App`	ChatLTV interface embedded in course Slack.	`Student`, `LangChain Middleware`
`LangChain Middleware`	Coordinates retrieval and LLM calls.	`Pinecone Vector DB`, `Azure OpenAI GPT-4`
`Pinecone Vector DB`	Stores embedded course corpus chunks.	`LangChain Middleware`, `Course Corpus`
`Course Corpus`	Cases, notes, books, slides, spreadsheets, Slack Q&A, and public material.	`Pinecone Vector DB`, `CMS`
`Azure OpenAI GPT-4`	Privacy-compliant LLM endpoint for answer generation.	`LangChain Middleware`
`CMS`	Faculty content-management and query-observation system.	`Course Corpus`

Data flow

A student asks ChatLTV privately or publicly in Slack. The system retrieves relevant course chunks from Pinecone, sends the question and context through Azure OpenAI GPT-4 via LangChain, returns an answer with document sources in Slack, and records queries in the CMS so faculty can inspect student needs.

“The relevant context was retrieved from the corpus, which was stored in a vector database (in our case, we chose Pinecone).”article, stated

“In addition to providing the answers through Slack, the LLM shared the document sources for the answers in the Slack reply”article, stated

Why this architecture

Slack fit the student workflow, RAG let ChatLTV use course-specific materials, Azure OpenAI addressed copyright/privacy constraints, and a CMS made the chatbot useful to faculty by exposing student queries and supporting admin content.

“This use of Slack had several benefits: first, we have three years' worth of Q&A content from the course Slack, and, second, Slack is emphasized as a part of the student workflow.”article, stated

“Leveraging Azure allows us to take advantage of the security, privacy, and compliance benefits, as well as guarantee that the data fed into the service is not used to retrain models that are then made available to others.”article, stated

Models used: Azure OpenAI GPT-4 · OpenAI GPT-4

Scale

The course had roughly 250 students across three sections.
“This year, there were three sections of the course with over 80 Harvard MBA students each so a total of roughly 250 students. The course is taught using the case method and has a fair amount of analytical work associated with it.”article, stated
The corpus contained roughly 200 documents and 15 million words.
“was trained on the entire corpus of my course -- including all the case studies, teaching notes, books, blog posts, and historical Slack Q&A mentioned above -- as well as selected and curated publicly available material. In total, the corpus contained roughly 200 documents and 15 million words.”article, stated
The backend was 8,000 lines, with 800 lines for RAG and 900 for content indexing.
“The total ChatLTV code base was 8000 lines for the backend (including 800 lines for RAG and 900 lines for content indexing”article, stated
About 170 students made over 3,000 queries.
“Over half our students -- roughly 170 -- made over 3000 queries over the course of the semester of ChatLTV. The course has 28 sessions, including 24 cases (versus exercises).”article, stated
Nearly 40% of users rated quality 4 or 5.
“When surveyed, nearly 40% of the students who used the chatbot gave it a quality score of a "4" or "5". The usage and quality were frankly higher than I had anticipated.”article, stated

Alternatives considered (roads not taken)

Direct OpenAI APIs.

Rejected because: HBS copyright and privacy constraints required stronger guarantees.

“Rather than using the OpenAI APIs directly, we used Microsoft's Azure OpenAI Service for both development and production use.”article, stated

Generic ChatGPT only.

Rejected because: Students needed course-specific answers and sources from the LTV corpus.

“We also made ChatGPT4 publicly available to the students in the course Slack alongside ChatLTV. That way, students could use the public chatbot or the course chatbot, depending on their needs.”article, stated

No admin-content repository.

Rejected because: Testing showed students asked administrative questions the course corpus did not answer cleanly.

“allowed us to play around with our prompts (i.e., prompt engineering) and content indexing. It also revealed an important feature that we later added: a repository of admin content. We realized in our informal user testing (i.e., discussions with prospective users, drawing from numerous students”article, stated

Patterns

`rag-retrieval` (education)

ChatLTV retrieves course-specific chunks from Pinecone before generating answers.

“This technique is known as Retrieval Augmented Generation (RAG)”article, stated

`privacy-preserving-processing` (compliance)

The deployment used Azure OpenAI and SOC2 Pinecone to protect copyrighted course content.

“guarantee that the data fed into the service is not used to retrain models”article, stated

`eval-driven-development` (quality)

The team used roughly 500 Q&A queries, manual testing, and automated OpenAI comparison to tune prompts and indexing.

“We created numerous test data sets (roughly 500 Q&A queries) to manually test ChatLTV”article, stated

Tradeoffs

Deployment model

Chose Azure OpenAI for development and production over Direct OpenAI APIs.

Rationale: Azure addressed security, privacy, compliance, and no-retraining requirements.

“used Microsoft's Azure OpenAI Service for both development and production use.”article, stated

Interaction channel

Chose Slack app over Separate course website chatbot.

Rationale: Students already used Slack for course reflections and materials.

“we have a course Slack and require our students to post reflections in Slack as part of their grade”article, stated

Gotchas (what bit them)

Course bots need administrative knowledge as well as conceptual knowledge.

Root cause: Students ask due dates and office-hours questions in addition to case-analysis questions.

Resolution: Add a Course Admin repository and train the model to review it first.

“students would want to ask administrative questions about the course”article, stated

Private usage dominates even in a Slack-native class.

Root cause: Students did not want peers to see preparation questions.

Resolution: Support private interactions visible only to student and faculty.

“99% of our students elected for private queries with the chatbot”article, stated

Evaluation

Harvard evaluated ChatLTV with roughly 500 Q&A test queries, manual testing, automated OpenAI comparison against ground truth, student usage, student quality ratings, and faculty inspection of query logs.

Metrics: Q&A test set quality · Manual testing · Automated quality score · Queries per case · Student quality ratings · Private/public query mix

“We also ran an automated evaluation by using OpenAI to compare the outputs to the ground truth data and generate a quality score”article, stated

Outcomes

About 170 students made over 3,000 ChatLTV queries.
Nearly 40% rated quality 4 or 5.
Faculty used query logs to understand student preparation and confusion before class.

The source reports 250 students, 200 documents, 15 million words, 3,000 queries, 130 queries per case, 99% private usage, 8,000 backend lines, and 9,000 CMS lines.

“Each morning before class, I would inspect the admin CMS to see what queries had been made by what students”article, stated