Building Personalized Financial and Gardening Experiences with LLMs

bud_financial_/_scotts_miracle-gro · Finance · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Personalized customer experiences for regulated finance and consumer gardening

Bud Financial and Scotts Miracle-Gro both needed LLM-powered customer experiences that could move beyond generic search or recommendations. Bud needed conversational financial guidance that uses transactional context while staying explainable and compliant. Scotts needed a gardening assistant that combines conversation, semantic knowledge retrieval, product recommendations, and plant-image context without giving broad or harmful product options.

Constraints:

• Bud operates in a highly regulated financial-services setting where advice paths need explainability, transparent assumptions, deterministic messaging, and validation.
• Scotts product recommendations have to be uniquely contextualized because a broad list of options can be unhelpful or even detrimental to a customer's plants.
• Scotts depends on unstructured research, SME, blog, and article content whose freshness and governance are ongoing problems rather than one-time setup work.
• Both teams needed rapid iteration because generative AI product behavior is qualitative and changes quickly as tooling evolves.

Architecture

The panel describes two related Google Cloud architectures. Bud trains proprietary customer-understanding models on GKE, runs production services on GKE with a mix of TPUs, GPUs, and Intel compute-optimized machines, and uses Gemini through the Vertex AI API suite for conversational capabilities in its YZ financial agent. A custom orchestration and validation layer routes regulated journeys into deterministic response paths before public rollout, with internal customer-agent sandbox testing used to reduce risk. Scotts' My Scotty assistant uses Vertex AI Conversations for human-like multi-turn dialogue, Gen AI Playbooks/Agents for the agentic product-recommendation workflow, Vertex AI Search for semantic retrieval over gardening knowledge, and custom Vertex AI image-recognition models to add plant-photo context to the conversation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  bud-transactional-data-platform["Bud transactional data platform"] --> bud-proprietary-models["Bud proprietary models"]
  gke-training-and-production-platform["GKE training and production platform"] --> bud-proprietary-models
  gke-training-and-production-platform --> yz-financial-agent["YZ financial agent"]
  bud-proprietary-models --> yz-financial-agent
  gemini-via-vertex-ai-apis["Gemini via Vertex AI APIs"] --> yz-financial-agent
  yz-financial-agent --> deterministic-validation-layer["Deterministic validation layer"]
  deterministic-validation-layer --> bank-app-conversations["Bank app conversations"]
  bank-app-conversations --> internal-customer-agent-sandbox["Internal customer-agent sandbox"]
  my-scotty-conversation-interface["My Scotty conversation interface"] --> gen-ai-playbooks-and-agents["Gen AI Playbooks and Agents"]
  gen-ai-playbooks-and-agents --> product-recommendation-path["Product recommendation path"]
  product-recommendation-path --> gardening-customer-advice["Gardening customer advice"]
  my-scotty-conversation-interface --> vertex-ai-search["Vertex AI Search"]
  vertex-ai-search --> scotts-knowledge-base["Scotts knowledge base"]
  scotts-knowledge-base --> unstructured-metadata-pipeline["Unstructured metadata pipeline"]
  unstructured-metadata-pipeline --> scotts-knowledge-base
  vertex-ai-search --> gardening-customer-advice
  custom-image-recognition["Custom image recognition"] --> plant-image-context["Plant image context"]
  plant-image-context --> my-scotty-conversation-interface

Components

Data flow

Bud starts from transactional data, trains customer-understanding models, exposes that context to the YZ conversational agent through Gemini and Vertex AI APIs, and validates regulated journeys so specific compliant messages are produced before a bank customer sees them. Scotts starts from user gardening questions, location/plant context, and optional plant images; My Scotty uses conversation and playbook/agent logic to gather context, sends knowledge questions to Vertex AI Search over Scotts' article base, feeds image-recognition context into the conversation, and returns either generalized gardening advice with citations or a single contextual product recommendation.

Why this architecture

The architecture separates model capability, domain context, and policy-sensitive orchestration. Bud keeps transactional-model training and production serving on GKE so data scientists can train proprietary models while production can balance latency, cost, performance, and scale. Its regulated conversational path adds deterministic validation instead of sending raw context directly to an API. Scotts uses managed Vertex AI conversation, search, and agent tooling because its problem is less about training a new foundation model and more about integrating semantic retrieval, product-recommendation context, and image context quickly while hiding vector/RAG infrastructure complexity.

Models used: Gemini · Bud proprietary models · Vertex AI custom image recognition models · LLMs for metadata generation

Scale

• Bud trains proprietary models from scratch in an offline GKE environment.
  “we train our own models uh from scratch uh we do that in in sort of our offline data environment which runs on gke”transcript, stated
• Bud production serving uses a heterogeneous compute mix of TPUs, GPUs, and Intel compute-optimized machine types.
  “we're using a mix of sort of the the tpus gpus the Intel compute optimized uh machine types”transcript, stated
• Scotts' recommendation flow avoids typical five-to-ten-option recommendation lists.
  “streaming websites or e-commerce will say customers like you also like these maybe 5 to 10 options right that's not something that we can do at Scots”transcript, stated
• Bud frames personalized outreach as more than twice as likely to receive a response.
  “if you get a personalized message and and Outreach um you're more than twice as likely to respond to that”transcript, stated
• Scotts' knowledge source spans research and SME-generated articles accumulated over multiple years.
  “knowledge based articles uh that Scots has generated through their research through their smes for over multiple years”transcript, stated

Alternatives considered (roads not taken)

Generic bank push notifications without real customer understanding.

Rejected because: The Bud example shows superficial personalization creating frustration when the bank offers an overdraft increase and then rejects the customer under lending policy.

Throw data directly at an LLM API for regulated financial journeys.

Rejected because: Bud says regulated paths require deterministic routing, explicit validation, and specific wording rather than raw API calls.

Keyword search over Scotts' gardening articles.

Rejected because: Scotts chose semantic Vertex AI Search so customers can ask naturally and still receive related results even without exact keywords.

Standard broad ecommerce-style product recommendations.

Rejected because: Scotts rejected broad lists because several options may not help and one may harm the plant; it needs a unique, contextual recommendation.

Patterns

safety-policy-enforcement (compliance)

Bud routes regulated conversational journeys through deterministic response and validation logic so financial advice paths stay explainable and compliant.

rag-retrieval (retrieval)

Scotts uses Vertex AI Search over its gardening knowledge base to return summaries, citations, and advice from existing articles and research.

hybrid-search (retrieval)

The gardening assistant moves beyond keyword matching by representing user queries in embedding space so natural questions can retrieve semantically related articles.

tool-use-orchestration (agent_architecture)

My Scotty combines conversation, agent/playbook routing, search, and image-recognition context to choose between advice and product-recommendation paths.

fine-tuning-adaptation (model_adaptation)

Bud trains its own models from transactional data and serves them alongside Gemini-powered conversational capabilities.

eval-driven-development (evaluation)

Bud tests the YZ agent internally with customer agents before public rollout to validate behavior against risk appetite.

data-quality-loop (data_governance)

Scotts treats unstructured content governance as ongoing work and uses LLMs to add metadata so advice stays accurate and current.

Tradeoffs

Use transactional customer understanding plus explainability for financial personalization.

Chose Contextual, explainable financial guidance over Surface-level push-notification personalization.

Rationale: Bud argues that financial personalization without actual customer understanding can frustrate customers and create compliance problems.

Keep Bud model training and production serving on GKE.

Chose GKE-backed proprietary model training and serving over A less flexible single-purpose serving environment.

Rationale: GKE gives Bud access to accelerators for training while production can balance latency, cost, performance, scaling, and management.

Move from UI-first generative tools toward APIs as the Bud implementation matured.

Chose API-based flexibility after rapid UI iteration over Staying only in managed UI workflows.

Rationale: The UI tools helped Bud iterate quickly, but API integration became more appropriate once the team needed custom orchestration and flexibility.

Use semantic search for gardening advice.

Chose Vertex AI Search with embeddings, summaries, and citations over Traditional keyword search over blogs and articles.

Rationale: Customers can describe gardening problems naturally and still retrieve meaningful advice even without the exact article keywords.

Prefer a single contextual product recommendation for Scotts.

Chose Specific agent-mediated recommendation over Five-to-ten generic recommendation options.

Rationale: Physical gardening products have context-sensitive consequences, so the system must gather user and plant context before recommending.

Use custom image recognition as conversation context rather than relying on LLM multimodality alone.

Chose Custom Vertex AI image-recognition models feeding the conversational interface over Treating plant photos as generic LLM multimodal input.

Rationale: The panel explicitly separates My Scotty's image path from typical LLM multimodality and uses custom plant-image recognition to enrich the conversation.

Gotchas (what bit them)

Superficial personalization can actively damage trust.

Root cause: A customer-facing AI or recommendation journey that does not understand eligibility or context can interrupt the customer with an offer and then immediately deny it.

Resolution: Bud centers the system on transactional customer understanding and explainability instead of generic bank-app outreach.

Regulated financial conversations cannot be treated as unconstrained generation.

Root cause: Financial recommendations depend on assumptions about income, affordability, and product eligibility that must be explained and validated.

Resolution: Use explainability, deterministic journey routing, specific compliant wording, and response validation around the LLM API path.

Generic recommendation-list patterns can be harmful in physical-product domains.

Root cause: A gardening recommendation can affect a living plant, so several plausible products are not equivalent options.

Resolution: Use an agentic workflow to gather enough context for a specific, well-contextualized product recommendation.

Unstructured knowledge governance remains a live operational problem.

Root cause: Scotts' advice depends on research and SME-authored articles, but governance practices for unstructured data are less mature than structured-data governance.

Resolution: Treat governance as ongoing work and use LLM-generated metadata to help keep research-backed advice accurate and up to date.

Managed platform names and interfaces changed during implementation.

Root cause: The Google generative AI tooling moved through names such as Gen App Builder, Vertex Conversation, and Agent Builder.

Resolution: Bud started with UI tools for rapid iteration, then moved toward APIs as flexibility needs grew.

Evaluation

Bud evaluates the YZ agent through user feedback, conversation analysis, deterministic journey checks, validation-logic checks, and internal customer-agent sandbox testing before public exposure. Scotts' retrieval path is described qualitatively through relevance, summaries, citations, natural-language query handling, and integration simplicity rather than formal benchmark metrics.

Metrics: User feedback on conversational experiences · Analysis of conversational behavior · Deterministic journey correctness · Validation logic working as intended · Risk appetite fit before public rollout · Semantic relevance and citation quality for gardening search · Personalized outreach response rate

Outcomes

• Bud built YZ, a bank-app conversational agent that uses customer context such as income and spending to personalize financial conversations.
• Scotts built My Scotty, a multi-turn gardening assistant for product recommendations and generalized advice.
• Scotts uses Vertex AI Search to let customers ask natural questions and receive meaningful summaries with citations.
• Scotts reports that Google technologies accelerated development of more relevant and personalized consumer products.
• Bud reports that personalized outreach is more than twice as likely to receive a response, while lack of personalization damages trust.

The source gives qualitative production architecture and a few concrete scale signals: Bud runs offline model training and production on GKE with GPUs, TPUs, and compute-optimized machines; Scotts avoids broad five-to-ten recommendation lists; Scotts' knowledge base spans research and SME articles over multiple years; and Bud cites more than twice the response rate for personalized outreach. It does not provide cost, latency, accuracy, or adoption-rate measurements.