Building Goal-Oriented Retrieval Agents for Low-Latency Recommendations at Scale

faber_labs · E-commerce · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Real-time recommendation and KPI optimization

Faber Labs built GORA to personalize rankings for businesses whose goals differ across e-commerce, healthcare, and financial services. The system needed to optimize subjective relevance toward measurable outcomes, adapt to in-session feedback, handle privacy-sensitive client data, serve recommendations with conversation-aware low latency, and avoid conflicting objectives across independently optimized model components.

Constraints:

• Different clients optimize for different measurable KPIs such as conversion rate, average order value, readmission reduction, or call-center volume.
• Real-time recommendation has to incorporate historical behavior, contextual insights, and in-session feedback without breaking privacy expectations.
• The system must run at low latency even when it is conversation-aware rather than a single-pass query-embedding retriever.
• Client data is messy and privacy-sensitive, especially in medical and financial contexts.
• Stacked ranking models can optimize against conflicting objectives unless trained toward a unified goal.

Architecture

GORA combines custom large event models, open-source LLM components, end-to-end reinforcement learning, and a Rust serving layer. Client event data trains large event models that generalize over user event sequences. A unified goal layer defines each client's measurable objective, then reinforcement learning jointly optimizes embedding generators, reranking models, and agent models toward that outcome. A real-time feedback processor updates context selection and agent behavior during conversations, while a Rust backend orchestrates user requests, model management, and GPU key-value caching for follow-up prompts. On-prem deployment and privacy-preserving cross-client learning address regulated clients.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  client-application[Client application] --> gora-serving-api[GORA serving API]
  gora-serving-api --> rust-backend[Rust backend]
  gora-serving-api --> real-time-feedback-processor[Real-time feedback processor]
  client-event-data[Client event data] --> large-event-models[Large event models]
  client-event-data --> privacy-preserving-learning-layer[Privacy-preserving learning layer]
  privacy-preserving-learning-layer --> large-event-models
  rust-backend --> large-event-models
  rust-backend --> gpu-key-value-cache[GPU key-value cache]
  rust-backend --> open-source-llm-components[Open-source LLM components]
  rust-backend --> real-time-feedback-processor
  large-event-models --> embedding-generators[Embedding generators]
  large-event-models --> reranking-models[Reranking models]
  large-event-models --> agent-models[Agent models]
  embedding-generators --> unified-reward-optimizer[Unified reward optimizer]
  reranking-models --> unified-reward-optimizer
  agent-models --> unified-reward-optimizer
  real-time-feedback-processor --> agent-models
  on-prem-deployment[On-prem deployment] --> rust-backend

Components

Data flow

A client defines a measurable KPI such as conversion, average order value, readmission reduction, or call-center reduction. GORA ingests that client's event data, including messy real-world behavior logs, and trains large event models from scratch. During an interaction, real-time user feedback influences context selection and agent behavior. Embedding generators, reranking models, and agent models are jointly optimized through a reinforcement-learning reward layer, then served through a Rust backend that manages request orchestration, model state, and GPU key-value caching for conversational follow-ups. Open-source LLMs glue results together and present them to users, while privacy controls isolate client data and support on-prem deployments.

Why this architecture

Faber split ranking from language generation because LLMs are not the core ranking engine; large event models handle event-sequence relevance while LLMs present or glue outputs. The unified reinforcement-learning objective prevents each model stage from optimizing a different metric. Rust was chosen for latency, concurrency, security, and cost reasons, while GPU KV caching addresses growing conversation state. On-prem deployment and cross-client privacy controls let the same architecture serve healthcare and finance without directly sharing client data.

Models used: Large event models · open-source LLMs · embedding generators · reranking models · agent models · policy models

Scale

• Amazon's recommendation engine is cited as contributing about 35 percent of revenue.
  “they use the recommendation engine to personalize user experiences and this actually contributes to a big chunk of the revenue about 35% of its revenue”transcript, stated
• GORA targets many marketplaces, retailers, neo-banks, medical clients, and other consumer-facing businesses.
  “scale it for many different marketplaces and retailers and even Neo Banks acting as an embedded kpi optimization layer for any consumer facing business”transcript, stated
• The system is designed for sub-second interactive rankings.
  “we are able to provide subsec results and interactive rankings uh to clinicians”transcript, stated
• The speaker cites 53 percent mobile abandonment after longer-than-three-second load times.
  “about 53% of mobile users abandon sites that take longer than 3 seconds to load”transcript, stated
• The company reports consistently above 200 percent joint increases in conversion rate and average order value.
  “we're actually seeing consistently above 200% increases jointly in in CBR and in conversion rate and average order value”transcript, stated

Alternatives considered (roads not taken)

Optimize stacked model components independently.

Rejected because: Independently optimized embedding, reranking, and agent models can work toward conflicting objectives; Faber instead jointly optimizes around the final KPI.

Use LLMs as the core ranking engine.

Rejected because: Faber uses open-source LLMs for language-facing glue, but relies on large event models for the ranking workload.

Use a Python-oriented backend for the production serving layer.

Rejected because: The team chose Rust despite a painful transition because performance, privacy/security, concurrency, and cost mattered for low-latency serving.

Regenerate conversation key-value cache for each follow-up.

Rejected because: Growing conversations become operationally unwieldy, so GPU-side intelligent caching avoids recomputing KV state.

Require clean, heavily preprocessed client data.

Rejected because: Faber considered messy client data inevitable and built the large event model pipeline to consume it directly for scalability.

Patterns

retrieval-ranking (retrieval)

GORA is a relevance-ranking system that reranks user-facing options toward client-defined KPIs such as conversion, order value, or readmission reduction.

data-quality-loop (feedback)

The system processes in-session user feedback and uses it to influence context selection, modify agent behavior, and optimize the overall system.

latency-optimization (infrastructure)

Faber uses Rust, component latency budgets, parallel processing, and GPU key-value caching to keep a conversation-aware ranking system responsive.

cost-aware-routing (operations)

The bootstrapped team chose a Rust backend partly to reduce infrastructure costs while serving low-latency model workloads.

privacy-preserving-processing (security)

GORA supports on-prem deployments and cross-client learning without direct information leakage between clients.

fine-tuning-adaptation (model-development)

Faber trains custom large event models from scratch on client event data rather than using a generic model for all domains.

semantic-caching (performance)

The system caches GPU key-value state for conversation follow-ups, avoiding repeated generation of prompt cache state.

Tradeoffs

Optimize all ranking components toward one client-defined goal.

Chose end-to-end reinforcement learning with unified goals over piecemeal optimization of stacked models.

Rationale: Client KPIs can conflict, so jointly optimizing embedding generators, rerankers, and agent models reduces objective mismatch.

Use large event models for ranking and LLMs for language glue.

Chose hybrid specialized-model plus LLM architecture over LLM-only recommendation logic.

Rationale: The core ranking task is handled by models trained for event data, while open-source LLMs are reserved for components where language presentation is useful.

Build the backend in Rust.

Chose Rust backend over more familiar Python/data-science stack.

Rationale: Rust imposed a painful learning curve but delivered memory safety, concurrency, lower costs, privacy/security benefits, and low-latency serving.

Cache conversation KV state on GPU.

Chose GPU key-value caching over regenerating cache for follow-up prompts.

Rationale: Conversation-aware systems accumulate large context state, and caching keeps follow-up latency and infrastructure overhead manageable.

Offer on-prem deployments.

Chose on-prem option for sensitive clients over cloud-only deployment.

Rationale: Healthcare and financial services clients require stronger data-control guarantees, even though on-prem support adds engineering burden.

Gotchas (what bit them)

Personalization and real-time feedback can conflict with privacy.

Root cause: The system needs user behavior and in-session signals, but clients in healthcare and finance handle sensitive data.

Resolution: Faber built large event models, privacy-preserving cross-client learning, and on-prem deployment options.

Stacked relevance models can optimize against each other.

Root cause: Metrics such as conversion rate and gross merchandise sales can move in opposite directions.

Resolution: The architecture jointly optimizes embedding generators, rerankers, and agent models against a unified goal.

Conversation-aware recommendation state grows operationally unwieldy.

Root cause: Follow-up prompts require representing an expanding conversation in the model.

Resolution: The team caches key-value state on GPU to reduce repeated computation for follow-up prompts.

Rust can impose a steep learning and migration cost for Python-heavy ML teams.

Root cause: Rust concepts are counterintuitive for practitioners coming from Python or data-science backgrounds.

Resolution: The team accepted the painful transition because performance, security, and cost gains were worth it.

On-prem deployment can be a heavy burden for early-stage companies.

Root cause: Privacy-sensitive clients need local deployment, but offering on-prem solutions adds engineering complexity.

Resolution: Faber argues the lightweight Rust system made on-prem deployment manageable for medical and banking clients.

Evaluation

Faber evaluates GORA against client business KPIs and latency requirements rather than only model-internal metrics. The talk emphasizes conversion rate, average order value, readmission-rate reduction goals, response/load times, comparison to conversational adaptive agent systems, and whether technical improvements translate into client business impact.

Metrics: conversion rate · average order value · readmission rates · call center volume · response times · load times · component latency budgets · user engagement · joint KPI improvement

Outcomes

• Faber reports sub-second interactive rankings for e-commerce and medical recommendation use cases.
• The company claims consistently above 200 percent joint increases in conversion rate and average order value.
• The Rust backend reportedly improved benchmarks, privacy/security, efficiency, and costs.
• GPU key-value caching helped manage conversational follow-up latency at scale.
• The architecture supports on-prem deployments for medical and banking clients.
• The reward layer reportedly learns across clients without direct private-information leakage.

The talk cites 35 percent of Amazon revenue from recommendations, sub-second results, 53 percent mobile abandonment above three-second loads, component latency budgets, many clients across industries, above-200-percent reported KPI increases, client-data training, cross-client learning without direct leakage, and on-prem deployments.