AI Agent System for Automated Travel Itinerary Generation

aimpoint_digital · Consulting · 2024 · source Agentic: yes · promotional bias: med · source-backed · extracted with prompt 0.5.2 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Travel itinerary planning

Aimpoint Digital needed itineraries that avoid stale LLM knowledge and hallucinated places while reducing hours of manual trip research.

Constraints:

• recency
• hallucination
• personalization
• runtime latency

Architecture

Aimpoint’s travel agent takes a traveler profile and runs a compound RAG architecture on Databricks: the query is embedded and sent in parallel to places, restaurants, and events retrievers over Databricks AI Search indexes fed by Delta tables and change data feed updates. Retrieved candidates are synthesized by open-source LLMs on Provisioned Throughput Endpoints with guardrails. Evaluation uses MLflow metrics, LLM-as-judge rubrics, and DSPy prompt optimization to improve itinerary quality and format.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  traveler-profile-input["traveler-profile-input"]
  embedding-model["embedding-model"]
  places-rag["places-rag"]
  restaurants-rag["restaurants-rag"]
  events-rag["events-rag"]
  ai-search-indexes["ai-search-indexes"]
  delta-tables["delta-tables"]
  synthesis-llm["synthesis-llm"]
  guardrails["guardrails"]
  llm-judge-and-dspy["llm-judge-and-dspy"]
  itinerary-output["itinerary-output"]
  traveler-profile-input --> embedding-model
  embedding-model --> places-rag
  embedding-model --> restaurants-rag
  embedding-model --> events-rag
  delta-tables --> ai-search-indexes
  ai-search-indexes --> places-rag
  ai-search-indexes --> restaurants-rag
  ai-search-indexes --> events-rag
  places-rag --> synthesis-llm
  restaurants-rag --> synthesis-llm
  events-rag --> synthesis-llm
  synthesis-llm --> guardrails
  synthesis-llm --> itinerary-output
  itinerary-output --> llm-judge-and-dspy

Components

Data flow

A user provides destination, dates, purpose, companions, and budget. The system converts that profile into embeddings, retrieves matching places, restaurants, and events through three parallel RAG paths, chooses the number of items based on trip length, and then sends the retrieved recommendations to an LLM that writes a cohesive itinerary under Databricks guardrails. Evaluation and prompt optimization run around that pipeline using LLM-as-judge metrics and DSPy.

Why this architecture

The architecture exists because standalone LLMs have stale training data and hallucination risk, while travel planning needs current restaurants, events, opening hours, and preference fit. Parallel RAGs let the system retrieve category-specific recommendations, and the final LLM only synthesizes grounded candidates. Evaluation and DSPy are wrapped around the pipeline because prompt changes can have unintended output effects.

Models used: DBRX Instruct · Meta-Llama-3.1-405b-Instruct · LLM-as-a-Judge · DSPy

Scale

• Travel planning can involve more than five hours of research across roughly 270 pages starting 45 days before a trip.
  “travelers spend over 5 hours researching and planning trips. People often visit up to ~270 web pages before finalizing their trip activities, and this process can start as early as 45 days before the trip.”article, stated
• The target user experience was itinerary generation in 30 seconds or less.
  “Could we leverage GenAI to streamline this process and produce an itinerary in 30 seconds or less?”article, stated
• The data set started with roughly 500 Paris restaurants and was designed to scale toward nearly 50,000 citywide records.
  “We collected data on ~500 restaurants in Paris, with plans to scale to nearly 50,000 citywide.”article, stated
• Retrieval volume scales with trip length, averaging three places/events and three daily restaurants.
  “The number of retrieved matches corresponds to the length of the trip, as shorter trips require fewer activities, while longer trips require more activities. On average, our system is configured to retrieve three places or events and three restaurants daily”article, stated

Alternatives considered (roads not taken)

Standalone ChatGPT-style itinerary generation

Rejected because: A standalone LLM can recommend stale, closed, or hallucinated venues because it lacks current travel data.

A single places-only RAG

Rejected because: An itinerary also needs restaurants and destination events, so the team composed separate RAG paths for each category.

Patterns

rag-retrieval (retrieval)

The system uses updated vector databases to ground itineraries in recent place, restaurant, and event information.

multi-agent-orchestration (retrieval)

Three RAG components run in parallel for places, restaurants, and events.

Tradeoffs

How many retrieval systems to compose.

Chose parallel specialized RAGs over one monolithic travel retriever.

Rationale: Separate indexes match places, restaurants, and events but increase orchestration complexity.

How to improve itinerary prompt quality without manual prompt tinkering alone.

Chose Use DSPy with an LLM-as-judge and custom metric over ground-truth data. over Relying on manual prompt edits after observing bad generated itineraries..

Rationale: The source says prompt changes can cause unintended consequences, so the team used DSPy with judge-based scoring and a custom metric to optimize outputs.

Gotchas (what bit them)

Standalone LLM itineraries can recommend stale or closed places.

Root cause: Base models lack fresh local data and may hallucinate activities.

Resolution: Use nightly updated RAG indexes and retrieve current destination data before generation.

Evaluation

Aimpoint used DSPy prompt optimization and LLM-as-a-judge automated evaluation.

Metrics: ~30 second itinerary generation · stakeholder feedback

Outcomes

• The system aims to reduce travel planning from over five hours to about 30 seconds.

Databricks platform case; technical but promotional.