RAG-Powered Chatbot for Sports Team Roster Management

philadelphia_union · Other · 2024 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Sports roster compliance and front-office decision support

Philadelphia Union needed front-office staff to interpret complex MLS roster and salary-budget rules quickly enough to support transfer and roster decisions without manual legalistic rule lookup becoming a bottleneck.

Constraints:

• The source rules were extensive, legalistic MLS roster regulations rather than a simple FAQ.
• Answers needed to stay grounded in current roster-rule documents so the club could maintain compliance.
• The interface needed to be usable by front-office staff without a custom front-end build.
• New PDF rule documents needed to become available automatically through the RAG pipeline.

Architecture

Philadelphia Union built a Databricks-hosted RAG chatbot for roster-rule interpretation. MLS roster-rule PDFs are continuously loaded into Databricks Volumes, converted to text, embedded through Databricks Foundation Model APIs, indexed in AI Search, and retrieved as context for DBRX Instruct. Agent Bricks Custom Agents orchestrates the retrieval and generation chain, serves it through Model Serving, and connects it to a Databricks Apps chat UI for the front office.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["MLS roster-rule PDFs"] --> B["Databricks Volumes"]
  B --> C["Text extraction and embedding pipeline"]
  C --> D["AI Search"]
  C --> E["Databricks Foundation Model API"]
  E --> F["DBRX Instruct"]
  D --> G["Agent Bricks Custom Agents"]
  F --> G
  G --> H["Review app and evaluations"]
  G --> I["Model Serving endpoint"]
  I --> J["Databricks Apps chat UI"]
  J --> K["Front office staff"]

Components

Data flow

Roster-rule PDFs are loaded into Databricks Volumes, text is extracted, embeddings are generated through Foundation Model APIs, and AI Search indexes those embeddings for retrieval. A front-office query enters the chat UI, Agent Bricks Custom Agents retrieves relevant rule context from AI Search, DBRX Instruct generates a grounded interpretation, and the response is returned through a Databricks Model Serving endpoint.

Why this architecture

RAG fit the case because the hard part was precise interpretation of changing external rule documents, not broad open-domain generation. Databricks also reduced integration work by colocating governed document storage, embeddings, vector search, hosted models, agent orchestration, evaluation, serving, and a no-code UI on one platform.

Models used: DBRX Instruct · Databricks embedding models · Databricks Foundation Model API

Scale

• The system was built for a 2020 MLS Supporters' Shield-winning club's front-office roster-planning workflow.
  “turned to the Databricks Data Intelligence Platform to streamline decision-making.”article, stated
• The RAG model moved from development to deployment in days.
  “The Union’s data team developed and deployed their RAG model in just days.”article, stated
• The user-facing workflow targets interpretation in seconds rather than slower manual rule lookup.
  “enables zero-shot interpretation of roster regulations in seconds. This accelerates decision-making and saves valuable time”article, stated

Alternatives considered (roads not taken)

Manual interpretation of roster-rule PDFs by front-office staff

Rejected because: The rules were extensive and legalistic enough to slow decision-making.

Self-hosting model infrastructure

Rejected because: The Databricks Foundation Model API made hosted DBRX access available without operational model-hosting burden.

Custom front-end development before deployment

Rejected because: Databricks Apps enabled a no-code ChatGPT-like interface and faster deployment.

Patterns

rag-retrieval (retrieval)

The chatbot retrieves roster-rule context from AI Search before DBRX Instruct generates a rule interpretation.

eval-driven-development (quality)

The agent framework includes a review app and evaluations used to collect feedback and validate the chatbot before deployment.

observability-tracing (operations)

Trace logging and feedback capture support iteration on the RAG lifecycle after deployment.

tool-use-orchestration (orchestration)

Agent Bricks orchestrates search, model calls, review/evaluation, serving, and UI integration as a deployable chain.

Tradeoffs

Use a RAG chatbot over direct manual rule lookup.

Chose Conversational RAG over governed roster-rule PDFs over Manual navigation of roster composition documents.

Rationale: RAG preserved grounding in official documents while reducing the time cost of interpreting legalistic rules.

Use hosted Databricks model APIs rather than own model infrastructure.

Chose DBRX Instruct through the Databricks Foundation Model API over Self-managed LLM hosting.

Rationale: The hosted API reduced operational overhead and let the team focus on application assembly.

Use platform-provided UI and agent orchestration.

Chose Databricks Apps plus Agent Bricks Custom Agents over Building bespoke orchestration and a custom interface first.

Rationale: The platform path reduced time to deployment and kept evaluation, serving, and UI integration close together.

Gotchas (what bit them)

Legalistic domain rules slowed operational decisions.

Root cause: Roster composition rules were extensive and difficult to interpret quickly during front-office planning.

Resolution: Use RAG over authoritative roster-rule PDFs to give conversational rule interpretations in seconds.

Document freshness matters for rule interpretation.

Root cause: New roster-rule PDFs need to be incorporated without manual rebuilds.

Resolution: Set up continuous ingestion into Databricks Volumes before extraction, embedding, and indexing.

A RAG system can appear useful before it is reliable enough for operational use.

Root cause: Rule interpretation affects roster compliance, so unvalidated answers carry operational risk.

Resolution: Use the framework's review app, feedback collection, and built-in evaluations before front-office release.

Evaluation

The case describes human feedback collection, a review app, built-in evaluations, trace logging, and performance evaluation in the Agent Bricks workflow before availability to the front office.

Metrics: Human feedback through review app · Built-in evaluations · Trace logging · Performance evaluation

Outcomes

• Front-office staff gained a ChatGPT-like interface for roster-rule interpretation.
• The data team developed and deployed the RAG model in days.
• The system enabled zero-shot interpretation of regulations in seconds.
• The team reduced the need to host or manage model infrastructure.

The reported production scope is the Philadelphia Union front office and the MLS roster-rule corpus; the article reports days-to-deploy and seconds-to-interpret but does not provide user-count or query-volume metrics.