Building a Production Fantasy Football AI Assistant in 8 Weeks

nfl · Media & Entertainment · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Production fantasy-football decision assistant for NFL fans

The NFL and AWS needed to ship an NFL+ fantasy-football AI assistant in eight weeks that could give fast, analyst-approved, data-backed roster advice without hallucinating stats, answering out-of-scope betting questions, or failing during game-day traffic spikes.

Constraints:

• the team had no formal design or requirements and only about eight weeks before the regular season
• fantasy managers needed fast advice shortly before kickoff while injury reports and expert opinions changed
• answers had to use accurate structured NFL data rather than hallucinated player statistics
• the NFL brand required security, reliability, legal approval, AI Council approval, and strict scope control
• the agent had to access vast Next Gen Stats data without dumping hundreds of thousands of tokens into every question
• Sunday, Monday, and Thursday football traffic spikes required resilient latency and throughput

Architecture

The NFL and AWS built a fantasy-football agent on EKS using Amazon Bedrock models, Strands Agents for model-driven orchestration, S3-backed session memory, and MCP as a semantic data layer over NFL Next Gen Stats and RotoWire. The team built a Semantic Stats Dictionary and broader data-boundary tools so the agent could request the right domain data with fewer calls. Production hardening added a Bedrock fallback provider, circuit breaker, per-turn reasoning instrumentation, guardrails on data volume/scope, and simple sliding-window caching for heavy MCP tool-call results.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  nfl-pro-user-interface[NFL Pro user interface] --> fantasy-ai-agent[Fantasy AI agent]
  fantasy-ai-agent --> strands-agents-runtime[Strands Agents runtime]
  strands-agents-runtime --> amazon-bedrock-model[Amazon Bedrock model]
  fantasy-ai-agent --> s3-session-memory[S3 session memory]
  fantasy-ai-agent --> mcp-semantic-data-layer[MCP semantic data layer]
  mcp-semantic-data-layer --> next-gen-stats-apis[Next Gen Stats APIs]
  mcp-semantic-data-layer --> rotowire-apis[RotoWire APIs]
  mcp-semantic-data-layer --> semantic-stats-dictionary[Semantic Stats Dictionary]
  strands-agents-runtime --> bedrock-fallback-provider[Bedrock fallback provider]
  fantasy-ai-agent --> per-turn-observability[Per-turn observability]
  mcp-semantic-data-layer --> heavy-call-cache[Heavy-call cache]

Components

Data flow

A user asks a fantasy question in NFL Pro. The EKS-hosted Fantasy AI agent uses a Bedrock LLM through Strands Agents, loads session context, consults the Semantic Stats Dictionary, calls MCP tools to retrieve NFL Next Gen Stats and RotoWire data, reasons over the results, and streams an answer. During failures or throttling, a Bedrock fallback provider can route the request to a secondary model; observability records per-turn tool calls, parameters, and returned data.

Why this architecture

The team optimized for getting analyst-grade intelligence into production quickly. Strands avoided building agent plumbing, MCP separated reusable data logic from agent logic, the Semantic Stats Dictionary compressed domain context, broad data-boundary tools reduced loop count, and fallback/observability/caching handled production risk without delaying the launch.

Models used: Amazon Bedrock LLM · Anthropic frontier model family on Amazon Bedrock · larger model for LLM-assisted refinement · Strands Agents model-driven orchestration

Scale

• The team shipped from idea to fan-facing product in eight weeks.
  “'cause we literally went from an idea to a product for fans to use in just eight weeks. Today, we're gonna talk about how we leveraged a new series of APIs.”transcript, stated
• The product target was 90% directionally approved responses from fantasy experts.
  “And secondly, when we say 90% directionally approved, what we mean is, is that when we evaluated all of the answers, we wanted the expert, the Fantasy Expert that we have at Next Gen Stats to be able to say, "Yeah, that passes the Smell Test.”transcript, stated
• Initial answers needed to stream in less than five seconds.
  “And so, we needed initial question answers to be in less than five seconds, and Fantasy-rich, deep analysis in less than 30 seconds. That is our 95th percentile under 30 seconds of every one”transcript, stated
• Rich analysis needed to finish under 30 seconds at the 95th percentile.
  “and Fantasy-rich, deep analysis in less than 30 seconds. That is our 95th percentile under 30 seconds of every one of our answers so far, meeting that threshold.”transcript, stated
• The first month handled 10,000 questions with zero reported incidents.
  “In our first month of 10,000 questions, not one incident was reported from a user or through our logs. Now, I know it's always good to talk about a product, but why don't I show you exactly how it works?”transcript, stated
• The Semantic Stats Dictionary reduced initial token consumption by 70%.
  “Dropped our initial token consumption by 70%. Instead of pulling in vast, highly descriptive human-readable API specs, we brought in information in the language”transcript, stated
• A single poorly bounded question consumed 1.3 million tokens during testing.
  “That's 1.3 million tokens that that question consumed. Why did it consume 1.3 million tokens? It wasn't the answer. The answer was actually really good.”transcript, stated
• Caching the two most recent heavy MCP calls doubled throughput and reduced cost by 45%.
  “That's it, just caching the most two recent, heavy MCP tool call results increased throughput by two times. And, folks, this is after all of the other optimizations we've shown, reduced our cost by 45%.”transcript, stated

  “optimizations we've shown, reduced our cost by 45%. The answer isn't build a sliding window to manage cache points. The answer is when you're thinking about speed”transcript, stated

Alternatives considered (roads not taken)

Embed data access logic directly in the agent code with Python decorators

Rejected because: The data layer needed parameter filtering, business validation, and parallel processing; embedding all that in the agent would be hard to maintain and scale.

Provide rich API specs and all domain context to every question

Rejected because: That would dump hundreds of thousands of tokens, so the team passed only semantically relevant dictionary slices at runtime.

Create narrow tools for each major use case

Rejected because: Many narrow tools caused fragmented answers, more loops, and extra latency; broader tools by data boundary let the agent express richer intent.

Fail visibly on Bedrock capacity or throttling

Rejected because: Launch-day user experience and NFL Pro's brand mattered more than preserving single-model behavior, so the team added a fallback provider and circuit breaker.

Ship conversation history, league integrations, feedback loops, and custom preferences in v1

Rejected because: Those features did not prove job number one: analyst-grade advice, timing, and production readiness.

Patterns

tool-use-orchestration (agent-architecture)

The fantasy assistant uses model-driven orchestration to decide which stats tools to call and in what order for lineup, waiver, trade, and player questions.

rag-retrieval (domain-context)

A Semantic Stats Dictionary and MCP data layer retrieve only the domain context and stats relevant to the user's fantasy question.

fallback-degradation (resilience)

A Bedrock fallback provider intercepts throttling or quota failures and sends the request to a secondary model, with a circuit breaker to avoid repeatedly hitting an overwhelmed service.

observability-tracing (agent-operations)

The team added per-turn reasoning instrumentation to inspect tool calls, parameters, data returned, and token-heavy decision patterns before launch.

semantic-caching (latency-cost)

The system caches the two most recent heavy MCP tool-call results because fantasy users often ask follow-ups about the same player.

guardrails-io-filtering (safety-scope)

The assistant is constrained to fantasy-football analysis and uses structured NGS APIs to avoid hallucinated stats and out-of-bounds betting/liability answers.

Tradeoffs

Use Strands Agents instead of building all agent scaffolding

Chose Strands Agents framework over custom session, prompt, model, and fallback plumbing.

Rationale: The eight-week deadline made framework scaffolding more valuable than owning every low-level agent runtime detail.

Separate agent logic from data logic with MCP

Chose MCP semantic data layer over decorator-defined tools embedded in the agent codebase.

Rationale: Separating concerns made complex data validation, filtering, parallel processing, reuse, and independent scaling manageable.

Accept bimodal fallback behavior for launch

Chose fallback provider and circuit breaker over strict single-model behavior that might expose throttling.

Rationale: Launch-day user experience and brand protection were more important than avoiding a temporary anti-pattern.

Cache recent heavy tool calls with a simple sliding window

Chose two most recent heavy MCP calls over predictive cache allocation before observing real users.

Rationale: Simple caching captured the common follow-up pattern and produced throughput/cost wins without over-engineering unlaunched behavior.

Ship job-one intelligence before nice-to-have features

Chose analyst-grade fantasy advice, timing, resilience, and scope guardrails over conversation history, league integrations, feedback loops, and custom preferences in v1.

Rationale: The launch goal was proving core fantasy advice under real production constraints; extra personalization could wait.

Gotchas (what bit them)

Domain stats are not self-explanatory to a model.

Root cause: Next Gen Stats has hundreds of fields whose meaning changes by context, so raw names or rich API specs alone were either ambiguous or too large.

Resolution: The team built a Semantic Stats Dictionary from analyst reasoning and LLM-assisted refinement.

Narrow use-case tools caused fragmented answers and slow loops.

Root cause: Autonomous agents given many narrow tools and thoroughness instructions will make many small calls instead of forming a broad answer.

Resolution: The team consolidated tools around data boundaries and gave the agent broader parameters with compressed data.

A correct answer can still be operationally unacceptable.

Root cause: The model's instruction to be complex, thorough, defensible, and backed by data caused one waiver question to request stats for every wide receiver.

Resolution: Per-turn instrumentation exposed the behavior, and the team added guardrails on maximum data pulled and when it could be pulled.

Capacity errors were unacceptable for launch-day users.

Root cause: Sunday traffic and emergent token consumption could exceed planned Bedrock service quotas despite due diligence.

Resolution: The team added fallback routing to another Bedrock-hosted model family and a circuit breaker around overwhelmed primary service calls.

Caching was hard because user conversation patterns were unknown before launch.

Root cause: Fantasy users ask open-ended questions and refer back, but the team could not predict exact conversation paths in advance.

Resolution: They studied analyst/user question patterns and used a simple sliding window over the two heaviest recent MCP tool-call results.

Evaluation

The team evaluated with thousands of possible questions, analyst expert review, structured-stat correctness, UAT questions from NFL Pro analysts, latency thresholds, production logs, per-turn observability, incident counts, and load/traffic assumptions from NFL Pro.

Metrics: 90% directionally approved by fantasy experts · initial answers under five seconds · complex analysis under 30 seconds at P95 · zero incidents in first month · 10,000 questions in first month · 70% initial token reduction · 90% throttling-exception reduction on launch day · 2x throughput improvement from heavy-call caching · 45% cost reduction

Outcomes

• The NFL and AWS shipped the Fantasy AI Assistant to NFL+ users through NFL Pro in eight weeks.
• The assistant achieved 90% analyst-approved responses.
• Initial streamed responses were under five seconds and complex answers met a 30-second P95 target.
• The first month handled 10,000 questions with zero reported incidents.
• Semantic dictionary filtering cut initial token consumption by 70%.
• Fallback routing reduced launch-day throttling exceptions by 90%.
• Caching the two most recent heavy MCP calls doubled throughput and cut cost by 45%.

The product launched in eight weeks, used hundreds/thousands of analyst questions for evaluation, served 10,000 questions in month one with zero incidents, hit sub-five-second initial streaming, 30-second P95 full responses, 70% token reduction, 2x throughput improvement, and 45% cost reduction.