Policy Search and Response System Using LLMs in Higher Education

ndus · Education · 2024 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Higher education policy search, compliance, and data governance

NDUS needed to help staff across 11 institutions find relevant policies, state laws, contracts, procedures, and codes without hours of manual searching across multiple sites, while operating inside public-sector procurement, governance, and access-control constraints.

Constraints:

• NDUS includes 11 institutions governed by the State Board of Higher Education.
• The system serves about 80,000 students, faculty, and staff.
• Users needed to search thousands of internal policies and state laws for compliance work.
• Existing processes depended on manual search and institutional knowledge that was hard to share or retain.
• Public-sector procurement timelines made adding new software tools difficult.
• Data governance had to ensure the right people accessed the right data and models.
• Policy answers needed references, page numbers, and links for verification.

Architecture

NDUS used its existing Azure and Databricks environment to build a low-risk Policy Assistant. More than 3,000 public PDFs are contained and synthesized in Databricks; AI Search keeps data synchronized; users ask plain-English questions through an API; a Llama 2-backed application uses Databricks Foundation Model APIs to generate policy search and response results with references, page numbers, and links; Unity Catalog controls access to data and models; and MLflow supports local testing and application management.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  Policy_document_corpus["Policy document corpus"] --> Databricks_lakehouse["Databricks lakehouse"]
  Databricks_lakehouse --> AI_Search["AI Search"]
  Databricks_lakehouse --> Unity_Catalog["Unity Catalog"]
  Databricks_lakehouse --> Foundation_Model_APIs["Foundation Model APIs"]
  AI_Search --> Policy_Assistant_API["Policy Assistant API"]
  Policy_Assistant_API --> Foundation_Model_APIs
  Foundation_Model_APIs --> Llama_2_model["Llama 2 model"]
  Llama_2_model --> Policy_Assistant_API
  Unity_Catalog --> Foundation_Model_APIs
  MLflow["MLflow"] --> Foundation_Model_APIs
  Authorized_user_workflows["Authorized user workflows"] --> Policy_Assistant_API

Components

Data flow

Policy documents, state laws, contracts, procedures, and codes are centralized in Databricks. AI Search synchronizes the searchable index as source data changes. A user sends a plain-English query through the API; the LLM retrieves and synthesizes relevant policy content; the response includes references, page numbers, and links; Unity Catalog governs who can access the underlying data and models; MLflow supports testing and GenAI application workflows.

Why this architecture

NDUS selected an existing Azure/Databricks foundation because it avoided public-sector procurement delays and let a small team build a governed GenAI application without custom model-hosting complexity. The use case was intentionally low-risk but valuable: public policy search with citations, page numbers, and links rather than an unconstrained answer generator.

Models used: Llama 2 · open source LLMs · DBRX · Databricks Foundation Model APIs · domain-specific LLMs

Scale

• NDUS includes 11 institutions.
  “The North Dakota University System (NDUS) includes 11 institutions”article, stated
• NDUS has five community colleges, four regional universities, and two research universities.
  “The North Dakota University System (NDUS) has five community colleges, four regional universities and two research universities”article, stated
• The system serves approximately 80,000 students, faculty, and staff.
  “approximately 80,000 students, faculty and staff.”article, stated
• Policy Assistant synthesizes more than 3,000 public PDFs.
  “Over 3,000 public PDFs are now contained and synthesized in Databricks”article, stated
• The centralized platform reduced time to bring new insights to market from one year to six months.
  “data teams have reduced the time it takes to bring new insights to market from one year to six months.”article, stated
• The Policy Assistant was built in six months.
  “NDUS began its generative AI journey by building a low-risk, automated search and response application called “Policy Assistant” in six months.”article, stated
• Manual searches previously could take hours.
  “Finding what you need among all those texts can take hours”article, stated

Alternatives considered (roads not taken)

Continue manual policy search across sites and documents

Rejected because: Manual search took hours, forced users to restart searches, and relied on institutional knowledge that was hard to share.

Procure a new standalone software tool

Rejected because: Public-sector procurement timelines made new tools daunting, while Databricks already existed in NDUS's Azure environment.

Host custom models from the start

Rejected because: Foundation Model APIs let NDUS build GenAI applications quickly without custom deployment complexity.

Select an LLM by name recognition alone

Rejected because: NDUS tested open-source LLMs and prioritized performance, inference time, size, and cost before choosing Llama 2.

Patterns

rag-retrieval (retrieval)

Policy Assistant retrieves and synthesizes policy PDFs, returning answers with references, pages, and links.

safety-policy-enforcement (governance)

Unity Catalog controls access to data and AI assets so the right people access the right data and models.

eval-driven-development (evaluation)

NDUS tested open-source LLMs on performance, inference time, size, and cost, then uses MLflow and local tests for GenAI applications.

observability-tracing (operations)

The use of MLflow gives NDUS a structured way to test and run ML/GenAI applications rather than treating the assistant as an unmanaged script.

model-routing (model-ops)

NDUS evaluated open-source LLM options and selected Llama 2 while considering consolidation on DBRX.

Tradeoffs

Start with a low-risk policy assistant.

Chose automated search and response over public PDFs over higher-risk private or decision-making GenAI workflows first.

Rationale: Policy search provided immediate productivity value while keeping the initial GenAI journey bounded and auditable.

Use existing Databricks/Azure infrastructure.

Chose expand within an already-vetted data platform over procure and integrate a new AI vendor stack.

Rationale: The existing relationship reduced procurement friction and let NDUS move quickly.

Use Foundation Model APIs rather than custom model deployment.

Chose serverless foundation model access over managing custom LLM hosting.

Rationale: The team could focus on the application and governance rather than model-serving infrastructure.

Select Llama 2 after evaluating practical operating criteria.

Chose Llama 2 on Databricks over choosing a model without testing performance, inference time, size, and cost.

Rationale: NDUS prioritized production behavior and operational fit over model hype.

Gotchas (what bit them)

Institutional knowledge does not scale across a university system.

Root cause: NDUS relied on people and unique resources to know where policy answers lived and how to interpret them.

Resolution: Centralize policy documents in Databricks and expose natural-language search with references, page numbers, and links.

Public-sector procurement can slow AI adoption.

Root cause: Adding new tools required lengthy procurement, making net-new software adoption daunting.

Resolution: Use existing Azure and Databricks infrastructure to get GenAI tools immediately.

Policy search must stay current as documents change.

Root cause: Static indexes would create stale LLM answers for compliance-sensitive policy questions.

Resolution: Use AI Search automatic data synchronization so LLM outputs remain up to date.

Compliance use cases require verifiable answers, not just fluent summaries.

Root cause: Users need to know what the actual policy says and which regulations apply.

Resolution: Return references, page numbers, and direct links with generated policy answers.

Evaluation

NDUS evaluated open-source LLMs for performance, inference time, size, and cost, then used MLflow for local tests and GenAI application workflows. Policy Assistant outputs are also implicitly evaluated through verifiability: references, page numbers, and links let users audit answers.

Metrics: LLM performance · Inference time · Model size · Cost · Local test results · Time to bring new insights to market · Policy answer references, page numbers, and links · Manual search time reduction

Outcomes

• NDUS built Policy Assistant in six months.
• More than 3,000 public PDFs are synthesized in Databricks.
• Users can ask plain-English policy questions and receive results with references, page numbers, and links.
• Time to bring new insights to market fell from one year to six months.
• NDUS is expanding into internal audit reports, daily enrollment tracking, predictive enrollment forecasting, unstructured news data, and domain-specific LLMs.

The solution supports an 11-institution university system with approximately 80,000 students, faculty, and staff, and it centralizes more than 3,000 public PDFs for policy search.