Building Enterprise-Scale AI Applications with LangChain and LangSmith

rakuten · E-commerce · 2024 · source Agentic: yes · promotional bias: high · source-backed · extracted with prompt 0.5.5 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Enterprise LLM application platform for business clients and employees

Rakuten needed to build and scale LLM-powered products across a large, multi-business organization: customer-facing business-client tools for market analysis, support, and documentation Q&A, plus internal employee chatbot creation over company documentation. The challenge was to move quickly while preserving prompt reuse, evaluation discipline, vendor flexibility, cost/performance control, and enterprise data-handling guarantees.

Constraints:

• Rakuten spans 70+ businesses, so teams risk duplicating or siloing prompt and architecture work.
• The employee platform was intended for 32k employees, making cost/performance tradeoffs material.
• Large-scale use required visibility into what applications were doing and why.
• User data required enterprise assurances, including data staying in Rakuten's environment and development/production access separation.
• The source reports a productivity target but not achieved production impact metrics.

Architecture

Rakuten built a suite of LLM applications on LangChain and LangSmith. Rakuten AI for Business includes AI Analyst for market intelligence, AI Agent for self-serve marketplace support, and AI Librarian for document summarization and real-time Q&A. Internally, three engineers used LangChain OpenGPTs to stand up an employee chatbot platform over internal documentation in one week, with plans to roll it to 32k employees. LangChain provides chain/agent architectures and vendor abstraction; LangSmith provides observability, prompt sharing through Hub, custom Testing and Eval, and enterprise environment separation.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  rakuten-business-clients[Rakuten business clients] --> rakuten-ai-for-business[Rakuten AI for Business]
  rakuten-ai-for-business --> rakuten-ai-analyst[Rakuten AI Analyst]
  rakuten-ai-for-business --> rakuten-ai-agent[Rakuten AI Agent]
  rakuten-ai-for-business --> rakuten-ai-librarian[Rakuten AI Librarian]
  rakuten-ai-for-business --> langchain-framework[LangChain framework]
  rakuten-ai-for-business --> langsmith-platform[LangSmith platform]
  rakutenians[Rakutenians] --> opengpts-employee-platform[OpenGPTs employee platform]
  opengpts-employee-platform --> langchain-framework
  opengpts-employee-platform --> internal-documentation[Internal documentation]
  opengpts-employee-platform --> langsmith-platform
  langchain-framework --> rakuten-ai-for-business
  langchain-framework --> opengpts-employee-platform
  langchain-framework --> model-provider-options[Model/provider options]
  model-provider-options --> langchain-framework
  langsmith-platform --> langsmith-hub[LangSmith Hub]
  langsmith-platform --> langsmith-testing-and-eval[LangSmith Testing and Eval]
  langsmith-platform --> environment-separation[Environment separation]
  langsmith-hub --> langsmith-platform
  langsmith-testing-and-eval --> langsmith-platform
  environment-separation --> langsmith-platform

Components

Data flow

Business clients interact with Rakuten AI Analyst, Agent, or Librarian depending on whether they need market intelligence, self-serve support, or documentation Q&A. These applications use LangChain interaction patterns and agent/chain architectures, while LangSmith gives teams visibility into behavior and why outputs happen. Prompt approaches can be shared through LangSmith Hub, and candidate models or cognitive architectures can be tested with custom evaluation metrics. Internal teams use OpenGPTs to create chatbots over internal documentation, with data kept inside Rakuten's environment and development access separated from production.

Why this architecture

Rakuten chose LangChain and LangSmith to move quickly across several LLM application types while retaining enterprise controls. LangChain supplied reusable interaction patterns, chains, agents, OpenGPTs, and vendor abstraction; LangSmith supplied observability, prompt reuse, custom evals, and environment-level assurances. That combination let a large multi-business company avoid isolated team experimentation while keeping the platform flexible enough for 32k-employee scale.

Models used: LLMs · LangChain · LangSmith · LangChain OpenGPTs · LangSmith Hub · LangSmith Testing and Eval

Scale

• Rakuten has 70+ businesses across e-commerce, travel, digital content, fintech, communications, and more.
  “The company has 70+ businesses in fields such as e-commerce, travel, digital content, fintech, communications and more.”article, stated
• The employee chatbot platform was initially built by three engineers in one week.
  “It only took three engineers one week to get the initial platform up and running.”article, stated
• Rakuten intended to roll the employee platform to 32k employees.
  “intend to roll the product to 32k employees”article, stated
• Rakuten started building with LangChain in January 2023.
  “they first started building with the framework in January of 2023.”article, stated
• Rakuten planned to distribute AI for Business across merchants, hotels, retail stores, and local economies.
  “Rakuten plans to distribute Rakuten AI for Business across its customer base, focusing particularly on merchants, hotels, retail stores, and local economies”article, stated
• Rakuten's stated aim was to improve productivity by 20%.
  “with the aim to improve productivity by 20%.”article, stated

Alternatives considered (roads not taken)

Limit the employee chatbot platform to a subset of employees.

Rejected because: Rakuten wanted broad employee access, which made model/provider flexibility and cost/performance control important at 32k-employee scale.

Let each team independently discover good prompts and architectures.

Rejected because: At large-company scale, independent teams produce uneven approaches, so Rakuten used LangSmith Hub to distribute best prompts and promote collaboration.

Choose a fixed model/provider architecture.

Rejected because: Rakuten needed vendor-lock-in abstraction and the ability to control cost/performance tradeoffs as usage scaled.

Rely on intuition when choosing models, prompts, and cognitive architectures.

Rejected because: Rakuten wanted experiments with custom evaluation metrics and measurable comparisons across approaches.

Patterns

prompt-versioning (collaboration)

Rakuten used LangSmith Hub to distribute successful prompts across teams and reduce isolated prompt experimentation.

eval-driven-development (quality)

Rakuten used LangSmith Testing and Eval with custom metrics to compare models, cognitive architectures, and other approaches scientifically.

observability-tracing (operations)

LangSmith was adopted as Rakuten scaled beyond prototypes to harden applications and make behavior visible.

rag-retrieval (knowledge access)

AI Librarian and the OpenGPTs employee platform use documentation as the basis for summarization, Q&A, and knowledge management.

model-routing (platform flexibility)

Rakuten valued LangChain abstraction so the team could keep provider options open and manage cost/performance tradeoffs for a 32k-employee rollout.

privacy-preserving-processing (enterprise security)

LangSmith was used with enterprise assurances that data stays in Rakuten's environment and dev/prod access can be separated.

tool-use-orchestration (application architecture)

Rakuten used off-the-shelf chain and agent architectures from LangChain to build multiple client and employee LLM applications quickly.

Tradeoffs

Use LangChain OpenGPTs for the employee platform.

Chose OpenGPTs with flexible cognitive architecture and user-experience control. over A narrower fixed chatbot implementation..

Rationale: The team needed rapid initial delivery plus enough flexibility to support a 32k-employee rollout and changing cost/performance decisions.

Adopt LangSmith as the hardening and visibility layer.

Chose LangSmith for visibility, Hub, Testing and Eval, and enterprise assurances. over Continuing with framework-only development and ad hoc observability..

Rationale: Scaling to more users required visibility into behavior, prompt sharing, measurable experiments, and user-data controls.

Roll AI access broadly rather than gate it to a subset.

Chose A planned rollout to 32k employees. over Restricting the employee chatbot experience to a smaller group..

Rationale: Rakuten wanted broad employee enablement, accepting the added need for cost/performance optionality.

Use custom evaluation metrics.

Chose LangSmith Testing and Eval with Rakuten-specific metrics. over Generic measurements or intuition-only comparisons..

Rationale: The organization needed to test multiple models and cognitive architectures against its own business requirements.

Gotchas (what bit them)

Independent teams can rediscover uneven prompt approaches.

Root cause: Large companies often have many teams developing ideas separately, so good approaches do not automatically spread.

Resolution: Rakuten used LangSmith Hub to distribute the best prompts and promote collaboration.

Scaling LLM apps without observability leaves teams unable to explain behavior.

Root cause: As more users interact with applications, teams need to know what is happening and why.

Resolution: Rakuten adopted LangSmith to harden work and provide visibility.

Broad employee rollout makes model/provider lock-in risky.

Root cause: At 32k-employee scale, cost/performance tradeoffs become material and the team needs optionality.

Resolution: Rakuten used LangChain's abstraction layer to customize cognitive architecture and abstract away vendor lock-in.

Enterprise user data requires stricter environment controls than a generic SaaS workflow.

Root cause: Rakuten handles user data and needs development and production access boundaries.

Resolution: LangSmith provided data residency within Rakuten's environment and separate access for development and production workflows.

Evaluation

Rakuten used LangSmith Testing and Eval with custom evaluation metrics to run experiments on models, cognitive architectures, and related LLM application approaches. LangSmith Hub supported prompt comparison and reuse, while LangSmith visibility helped teams understand application behavior as usage scaled.

Metrics: Custom evaluation metrics for models and cognitive architectures. · Experiments across multiple approaches. · Prompt reuse and collaboration through LangSmith Hub. · Planned productivity improvement target of 20%.

Outcomes

• Rakuten AI for Business covered market analysis, customer support, and documentation Q&A use cases.
• Three engineers got the initial OpenGPTs employee chatbot platform running in one week.
• Rakuten intended to roll the employee platform to 32k employees.
• Rakuten planned to distribute AI for Business across merchants, hotels, retail stores, and local economies.
• The case reports a 20% productivity improvement aim, but not measured achieved impact.

Rakuten's stated scale is organizationally large: 70+ businesses, 32k employees, and a planned customer-base distribution. The source is a LangChain customer story, so results should be read as vendor-reported and partially aspirational.