Scaling AI-Powered Student Support Chatbots Across Campus

uc_santa_barbara · Education · 2022 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Education

UC Santa Barbara needed to help students navigate distributed campus services after COVID while reducing repetitive staff workload and preserving escalation for complex questions.

Constraints:

• students ask questions outside business hours
• student services are spread across departments and websites
• low-confidence answers should be handled carefully
• student language and expectations needed testing before launch

Architecture

UCSB deployed Gravity's Ivy/Ocelot chatbot platform as a campuswide support layer rather than isolated departmental bots. The rollout began with five high-impact departments, then grew to 19 bots. Department websites and curated question/answer libraries feed a shared-brain model where offices can own content, share selected answers with other virtual assistants, and add custom responses for local terminology. Students enter through pathway buttons, free-form chat, SMS, or optional live chat. High-confidence answers come from crawled website content and RAG/generative answer retrieval; low-confidence or sensitive cases route to links, intake forms, or staff follow-up. Analytics on chats, gaps, hot topics, and top questions feed website updates and bot training.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  department-websites["department-websites"] --> website-crawler["website-crawler"]
  website-crawler --> shared-brain-content["shared-brain-content"]
  shared-brain-content --> confidence-gated-answering["confidence-gated-answering"]
  pathway-buttons["pathway-buttons"] --> confidence-gated-answering
  confidence-gated-answering --> low-confidence-escalation["low-confidence-escalation"]
  low-confidence-escalation --> live-chat-intake["live-chat-intake"]
  sms-campaigns["sms-campaigns"] --> confidence-gated-answering
  confidence-gated-answering --> analytics-gap-loop["analytics-gap-loop"]
  analytics-gap-loop --> department-websites
  analytics-gap-loop --> department-admins["department-admins"]
  department-admins --> shared-brain-content
  department-admins --> live-chat-intake

Components

Data flow

A department joins a cohort after confirming its website and team are ready. Gravity crawls the website, UCSB staff train the bot's brain and shared terminology, and shared content libraries can be inherited by other virtual assistants. A student arrives through a bot button, free-form question, or SMS. The platform searches website/content-pack knowledge, uses RAG/generative AI when needed, returns an answer only above a confidence threshold, and otherwise guides the student to links or staff escalation. Chat analytics then shows missing/outdated content, common topics, and website updates that improve future answers.

Why this architecture

UCSB needed 24/7 answers without turning every department into a chatbot engineering team. The crawler/shared-brain setup lets offices own content while sharing common answers across campus. Confidence gating and optional escalation preserve staff involvement for complex or sensitive cases. The cohort rollout lets UCSB start with high-impact departments, learn student language, and expand the same campus resource without requiring live chat or SMS from every office on day one.

Scale

• UCSB serves about 25,000 students across over 200 majors and degrees.
  “it's about 25,000 students ranging across over 200 majors and degrees offered.”transcript, stated
• The pilot started with five departments.
  “So, we landed on on those five and those were our pilot departments.”transcript, stated
• The deployment grew to 19 bots, with three outside the original division.
  “we we are at 19 bots and again we we started with five and we are now we have three bots outside of our division.”transcript, stated
• The platform handled almost 40,000 chats over about two and a half years.
  “we're almost at 40,000 chats. I think it was like 39,700 and change when I looked last. So that's across two and a half years”transcript, stated
• About 30% of conversations happened outside business hours and on weekends.
  “30% of conversations are taking place outside of business hours and on weekends.”transcript, stated
• An SMS pilot to 75 unregistered students drove at least 30% sign-up/show-up within 48 hours.
  “let's get the 75 students who haven't signed up. Let's send them a really brief short text”transcript, stated

  “it was at least 30% of students who within those 48 hours signed up and showed up to that Monday program.”transcript, stated

Alternatives considered (roads not taken)

Traditional office-hours, phone, and email-only support

Rejected because: Students asked questions at 2am, on phones, and while crossing campus, so a 9-to-5 office model could not meet demand.

Chatbot options where UCSB had to build the bot brain itself

Rejected because: UCSB's departments had limited capacity, so website crawling and upfront vendor support were decisive.

Siloed departmental bots with no shared content

Rejected because: Students do not know or care which office owns the answer; shared brain content lets departments inherit answers while retaining local control.

Requiring every department to staff live chat from launch

Rejected because: Live chat was useful but optional; departments could launch chatbot support first and add live coverage later.

Patterns

rag-retrieval (retrieval)

The chatbot answers by crawling and using department websites as its knowledge base.

confidence-thresholding (quality-control)

The system answers only when confidence is high enough and can route uncertain questions differently.

human-in-the-loop-gate (support)

The project explicitly preserved routing to staff for next-level questions.

Tradeoffs

Use a crawler/shared-brain platform instead of making departments build chatbot knowledge from scratch.

Chose Ivy/Ocelot crawling plus shared content libraries over department-built bot brains for each office.

Rationale: The selected platform reduced the content-building burden and let departments share answers across virtual assistants.

Make live chat and intake forms optional by department.

Chose department-specific escalation choices over requiring live chat coverage from launch.

Rationale: Different offices had different staffing and risk profiles; health-related questions needed follow-up, while other bots could remain anonymous.

Answer only above confidence thresholds and route uncertainty.

Chose confidence-gated answers with low-confidence links/escalation over free-form bot responses regardless of certainty.

Rationale: The campus treated traceability and correction as a safety net for AI answers, especially when a bot response might be wrong or outdated.

Gotchas (what bit them)

Department staff could not validate chatbot language alone.

Root cause: Students phrase questions differently from staff and may expect answers from unexpected departments.

Resolution: UCSB required student testers before launch.

Evaluation

Student testers were used before launch, and the system reports accuracy/confidence behavior.

Metrics: 85% confidence threshold · 98% terminology accuracy

Outcomes

• UCSB grew from five bots to 19 bots and nearly 40,000 chats.
• About 30% of conversations happened outside business hours and on weekends.

The deployment expanded beyond one division toward campuswide use.