Evolving LLMOps Architecture for Enterprise Supplier Discovery

various · E-commerce · 2024 · source Agentic: yes · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise supplier discovery

Scoutbee needed to turn LLMs into a trustworthy supplier-discovery product for enterprise supply-chain teams, but early API-based chat suffered from hallucinated suppliers, weak domain understanding, privacy concerns, and hard-to-debug agent behavior.

Constraints:

• Supplier discovery requires domain-specific understanding that generic web search and foundational models did not provide.
• Enterprise customers were uncomfortable with production workloads and data moving through third-party model APIs.
• Hallucinated suppliers looked realistic enough to undermine trust.
• The business process was dynamic rather than a single linear query.
• Data provenance and explainability mattered to enterprise users evaluating suppliers.
• Data science teams needed infrastructure that worked across ML, LLM, and data workloads.

Architecture

Scoutbee evolved from a ChatGPT API prototype into a layered supplier-discovery architecture. The system added self-hosted/open-source LLM serving for privacy, domain-adapted agents and Graphs-of-Thought guardrails, RAG with Chain-of-Thought query decomposition over knowledge-graph data, citation-backed answers, observability over conversations and results, LLM-assisted data expansion, human validation, third-party data integration, and Ray/Anyscale as a universal compute layer for ML, LLM, and data workloads.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  supplier-discovery-user-interface["Supplier discovery user interface"] --> query-understanding-layer["Query understanding layer"]
  query-understanding-layer --> graph-of-thought-guardrails["Graph-of-Thought guardrails"]
  query-understanding-layer --> chain-of-thought-planner["Chain-of-Thought planner"]
  llm-api-layer["LLM API layer"] --> chain-of-thought-planner
  open-source-llm-serving["Open-source LLM serving"] --> llm-api-layer
  chain-of-thought-planner --> knowledge-graph["Knowledge graph"]
  data-expansion-pipeline["Data expansion pipeline"] --> knowledge-graph
  human-validation-loop["Human validation loop"] --> data-expansion-pipeline
  universal-compute-framework["Universal compute framework"] --> open-source-llm-serving
  universal-compute-framework --> data-expansion-pipeline
  knowledge-graph --> grounded-answer-composer["Grounded answer composer"]
  graph-of-thought-guardrails --> grounded-answer-composer
  grounded-answer-composer --> observability-and-product-metrics-store["Observability and product metrics store"]
  grounded-answer-composer --> supplier-discovery-user-interface

Components

Data flow

A user describes a supplier need, the system standardizes and may split the request, Chain-of-Thought prompting plans data calls, retrieval queries the knowledge graph and data partners, guardrails follow the user's business-process state, the LLM summarizes grounded supplier results with citations, and conversation/result telemetry feeds observability and product metrics.

Why this architecture

Scoutbee layered privacy, domain adaptation, guardrails, RAG, knowledge graphs, and scalable compute because each stage exposed a different production failure: generic models lacked domain knowledge, self-hosting increased complexity, agents were hard to debug, RAG reduced hallucinations, and knowledge graphs preserved provenance better than opaque embeddings for enterprise users.

Models used: ChatGPT API · LLaMA-13B · FastChat API · LangChain · Chain-of-Thought prompting · Graphs of Thought guardrails · Ragas · Knowledge graph · Ray · Anyscale

Scale

• The production journey covered roughly 16 to 18 months.
  “our journey through the last 16 to 18 months, and what it took for us to bring this into production”article, stated
• Scoutbee serves supplier discovery for large organizations such as Unilever and Walmart.
  “we help organizations such as Unilever, Walmart, and organizations of that size to do supplier discovery”article, stated
• Building the original knowledge-graph ontology took about 6 to 9 months.
  “What took us about 6 to 9 months of effort to build this is something that you can actually build for your domain in maybe a few months using an LLM.”article, stated
• LLM-generated training data plus human validation reduced annotation effort by 10x to 20x.
  “our efforts went down by 10x to 20x of what we would have to spend with just humans annotating the data”article, stated

Alternatives considered (roads not taken)

Use a simple ChatGPT API integration in production.

Rejected because: Generic models lacked supplier-domain knowledge, hallucinated suppliers, became too chatty, and raised privacy concerns for customers.

Fine-tune or retrain a foundational model for domain adaptation.

Rejected because: Full model training was too expensive and data-intensive compared with in-context/few-shot/agent-based adaptation.

Use open-ended agents as the main planner.

Rejected because: Agent behavior was difficult to test and debug, and agents sometimes skipped APIs or made up answers.

Represent supplier data primarily with embeddings.

Rejected because: Embeddings were faster for some questions but hard to explain and correct for enterprise users.

Build all compute-management infrastructure internally.

Rejected because: Ray/Anyscale was faster to operate, supported privacy on Scoutbee infrastructure, and avoided a long build path.

Patterns

rag-retrieval (hallucination reduction)

Scoutbee forced answers to use retrieved supplier data and citations from its knowledge graph rather than letting the foundational model invent suppliers.

guardrails-io-filtering (business-process safety)

Scoutbee modeled the supplier-discovery process as a graph and invoked guardrails depending on the user's process state.

query-rewriting (retrieval planning)

The system rewrote and split user stories into multiple retrieval queries before summarization.

synthetic-data-generation (data expansion)

Scoutbee used a superior LLM to generate high-quality training data for smaller models, with humans validating the result.

observability-tracing (product metrics)

Scoutbee stored conversations, shown results, and metadata so the team could inspect behavior and product metrics.

Tradeoffs

How to satisfy enterprise privacy while using LLMs.

Chose Self-hosted open-source LLM serving over A provider-only ChatGPT API path.

Rationale: Self-hosting increased cost and complexity, but it answered customer privacy concerns before the team over-invested in the product.

How to reduce hallucinations in supplier recommendations.

Chose RAG with Chain-of-Thought over knowledge-graph data over Open-ended agents answering from model knowledge.

Rationale: RAG reduced hallucinations and made testing easier because the team could inspect what the LLM was trying to do.

How to represent enterprise supplier data.

Chose Knowledge graph as system of record over Primarily vectorizing and embedding all data.

Rationale: The knowledge graph preserved domain relationships and provenance that enterprise users could understand and correct.

How to scale heterogeneous ML, LLM, and data workloads.

Chose Ray/Anyscale universal compute framework over Continuing separately with Spark-heavy pipelines and custom infrastructure.

Rationale: Ray gave data scientists Python-friendly scaling and reduced maintenance complexity while preserving on-infrastructure privacy.

Gotchas (what bit them)

Generic LLMs fabricated realistic-looking suppliers.

Root cause: The foundational model lacked supplier-discovery domain knowledge and was allowed to answer without grounding.

Resolution: Move to RAG over knowledge-graph data with citations and provenance.

Privacy concerns could invalidate the whole development path.

Root cause: Enterprise customers did not want production supplier-discovery workloads integrated with external provider APIs.

Resolution: Test whether open-source LLMs could be hosted by Scoutbee before proceeding further.

Agent debugging became a production blocker.

Root cause: The team could not inspect why agents called or skipped data APIs and sometimes made up answers.

Resolution: Replace open-ended agents with Chain-of-Thought/RAG flows that made the reasoning and retrieval path more visible.

Embedding-only approaches were hard to explain to enterprise users.

Root cause: Vector similarity could find answers but did not expose relationships and corrections as clearly as the knowledge graph.

Resolution: Use the knowledge graph as the system of record while continuing some embedding experiments.

Spark-based data pipelines were difficult for the data-science team to operate.

Root cause: Spark's Java/Scala ecosystem and cluster behavior were far from the Python scientific stack the team used.

Resolution: Introduce Ray as a universal compute framework for ML, LLM, and data workloads.

Evaluation

Scoutbee evaluated each architecture stage through customer feedback, hallucination behavior, agent debuggability, Ragas-style generation/retrieval scoring, stored conversation/result telemetry, and human validation of generated data.

Metrics: Customer feedback on usefulness and privacy · Hallucination reduction · Generation-vs-retrieval scores from Ragas · Stored conversations, result data, and product metrics · Human validation of generated training data

Outcomes

• Scoutbee moved from hallucinated supplier answers to citation-backed recommendations with data provenance.
• RAG reduced hallucinations and made testing easier by removing opaque agent behavior.
• Knowledge-graph expansion and third-party data improved answer coverage.
• LLM-assisted annotation plus human validation reduced training-data effort by 10x to 20x.
• Ray/Anyscale provided a common compute path for ML, LLM, and data workloads.

The architecture supports enterprise supplier discovery for large customers after an 18-month production journey, but the talk does not provide quantitative production traffic numbers.