Retrieval Augmented LLMs for Real-time CRM Account Linking

schneider_electric · Energy · 2023 · source Agentic: no · promotional bias: med · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: CRM account linking and corporate hierarchy resolution

Schneider Electric needed to link customer accounts to parent entities across CRM systems, but the workflow depended on recent acquisitions and organizational changes that static LLM knowledge could not reliably know.

Constraints:

• New customer accounts are added daily and previously required account teams to sort and link them manually.
• Corporate parent-subsidiary relationships can change through recent acquisitions, market news, and restructuring.
• The model needed current external knowledge rather than only pretrained model memory.
• Education and healthcare accounts required domain-specific parent-relationship prompts because ordinary parent-company wording did not generalize.
• Reliable corporate structure data could also appear in tabular SEC 10-K filings rather than only web pages.

Architecture

Schneider Electric and AWS built a RAG account-linking pipeline around Flan-T5-XXL on SageMaker JumpStart. A LangChain chain queries Google Serper for current parent-company evidence, combines retrieved snippets with a prompt, and sends the augmented question to the SageMaker endpoint. The pipeline was extended with a first-stage domain identification step that selects sector-specific prompts, and with SEC 10-K tabular data accessed through a pandas dataframe agent for more trustworthy corporate hierarchy evidence.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  A["CRM account records"] --> B["Google Serper retrieval chain"]
  A --> C["Domain classification RAG"]
  C --> D["Domain-specific prompt selector"]
  B --> E["Flan-T5-XXL SageMaker endpoint"]
  D --> E
  E --> F["Account-linking output"]
  E --> G["SEC 10-K dataframe agent"]
  G --> F
  F --> H["Downstream analytics teams"]

Components

Data flow

A CRM account name enters the pipeline. The system forms a parent-company query, retrieves current web evidence through Google Serper, packages the search context and question into a prompt, and asks the Flan-T5-XXL SageMaker endpoint for the relationship. For harder sectors, a domain RAG step classifies the account first so a healthcare or education-specific prompt can be used. SEC 10-K tables can be loaded into a pandas dataframe agent for direct parent/subsidiary lookup.

Why this architecture

A RAG architecture was necessary because the task depended on current corporate events. SageMaker JumpStart reduced deployment friction for Flan-T5-XXL, LangChain provided tool orchestration for web and table retrieval, and domain-specific prompts addressed cases where the same parent-company question had different semantics by sector.

Models used: Flan-T5-XXL · LangChain pandas dataframe agent

Scale

• Schneider Electric's customer base grows with new customers added daily.
  “As their customer base grows, new customers are added daily, and their account teams have to manually sort through these new customers”article, stated
• The motivating acquisition example happened in February 2023, after many models' knowledge cutoffs.
  “the Schneider Electric customer database) concerns an acquisition that happened in February 2023 and thus would not be caught by the LLM alone due to knowledge cutoffs.”article, stated
• Sector-specific prompting improved measured accuracy from 55% to 71%.
  “The sector specific prompts have boosted the overall performance from 55% to 71% of accuracy.”article, stated

Alternatives considered (roads not taken)

Use the pretrained LLM alone

Rejected because: The model would miss recent acquisitions and restructuring because its knowledge is limited by training date.

Use one blanket parent-company prompt

Rejected because: The same parent-company phrasing did not generalize to education and healthcare accounts.

Rely only on web search snippets

Rejected because: SEC 10-K filings provide a more reliable structured source for subsidiaries and subdivisions.

Patterns

rag-retrieval (retrieval)

The pipeline augments Flan-T5-XXL with Google Search results and SEC 10-K data so account-linking answers include current corporate evidence.

tool-use-orchestration (orchestration)

LangChain orchestrates calls to Google Serper, the SageMaker-hosted LLM, and a pandas dataframe agent over corporate-structure tables.

prompt-versioning (prompting)

The team moved from a blanket prompt to domain-specific prompt selection after discovering sector semantics changed the account-linking question.

eval-driven-development (evaluation)

Prompt changes were justified by measured accuracy gains, with sector-specific prompts raising performance from 55% to 71%.

Tradeoffs

Knowledge source for parent-company answers

Chose retrieval-augmented LLM with Google search over LLM-only question answering.

Rationale: Current acquisitions and restructuring are central to the task and may occur after model training.

Prompt design

Chose domain identification followed by sector-specific prompts over one generic parent-company prompt.

Rationale: Sector-specific prompts substantially improved measured accuracy and handled domains where parent-company wording was ambiguous.

Additional corporate-structure source

Chose SEC 10-K tabular augmentation over web search only.

Rationale: 10-K filings provide trustworthy structured subsidiary data that can be queried directly.

Gotchas (what bit them)

Static LLM knowledge was stale for account-linking decisions.

Root cause: Corporate relationships change after the model's training cutoff.

Resolution: Couple the LLM to Google search and SEC 10-K retrieval before asking it to infer parent relationships.

A generic parent-company question failed across sectors.

Root cause: Parent-company semantics are not meaningful in the same way for education or healthcare accounts.

Resolution: Add a first-step domain classifier and select a sector-specific prompt before asking for the parent relationship.

Corporate hierarchy evidence may be structured, not prose.

Root cause: Reliable subsidiary relationships can live in SEC 10-K tables rather than search-result snippets.

Resolution: Use LangChain's pandas dataframe agent to query tabular parent/subsidiary records.

Evaluation

The case reports accuracy improvement from domain-specific prompt selection and frames the pipeline by whether it reduces manual account-linking effort and improves downstream data access.

Metrics: Sector-specific prompts improved accuracy from 55% to 71% · Manual effort in account-linking workflows reduced · Faster data access for downstream analytics teams

Outcomes

• The RAG pipeline addressed LLM knowledge cutoffs for recent corporate events.
• Domain-specific prompt selection raised accuracy from 55% to 71%.
• SEC 10-K data created a path for more trustworthy subsidiary lookup.
• The architecture reduced manual linking effort and improved access to hierarchy data for analytics teams.

The workflow targets daily CRM account growth and parent-subsidiary data used across Schneider Electric's account-linking and analytics workflows.