Building Analytics Applications with LLMs for E-commerce Review Analysis

microsoft · E-commerce · 2023 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce review analytics

Microsoft's case study needed to turn unstructured product reviews into sentiment, aspect, and topic insights without maintaining many separate traditional ML models for each analytics task.

Constraints:

Traditional workflows require labeled datasets and separate models for sentiment, aspect extraction, and clustering.
Business stakeholders need explainability, not only opaque predictions.
Production use requires structured outputs and deterministic-enough behavior.
The work was an analytics application rather than an autonomous agent.

Architecture

The solution uses OpenAI completions with carefully engineered few-shot prompts to perform sentiment analysis, product-aspect extraction, and topic grouping over review data. Temperature is set to 0 for mostly deterministic outputs, and prompt instructions request formatted outputs for downstream analytics.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  reviews-api[Reviews API] --> prompted-llm-pipeline[Prompted LLM Pipeline]
  prompted-llm-pipeline --> openai-completion-model[OpenAI Completion Model]
  prompted-llm-pipeline --> structured-output[Structured Output]
  structured-output --> business-analytics[Business Analytics]

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Reviews API`	Source of e-commerce customer review text.	`Prompted LLM Pipeline`
`Prompted LLM Pipeline`	OpenAI completions workflow for analytics tasks.	`OpenAI Completion Model`, `Structured Output`
`OpenAI Completion Model`	LLM performing sentiment, aspect extraction, and clustering from prompts.	`Prompted LLM Pipeline`
`Structured Output`	Formatted analytics results for downstream processing.	`Business Analytics`
`Business Analytics`	Stakeholder analysis of sentiment, aspects, and topics.	`Structured Output`

Data flow

Reviews are extracted with a reviews API, passed to a prompt through the completions endpoint, and the LLM returns formatted sentiment, aspect, and topic outputs that can be analyzed downstream.

“How does one delve into such a dataset? Fortunately, the process is straightforward. In this article I show how, beginning with extracting reviews using a reviews API and followed by deploying Machine Learning models to identify product aspects and sentiment.”article, stated

“Once set up, execute the Python code via the completions endpoint. The meticulously crafted prompt in the code below instructs the model on multiple tasks while ensuring a properly formatted output for easier downstream processing.”article, stated

Why this architecture

The LLM prompt approach replaces multiple task-specific ML models with one flexible model, while few-shot prompting and temperature 0 make outputs more reusable and explainable for business analytics.

“For each task — whether sentiment analysis, product aspect extraction, or topic clustering — different models must be trained and maintained.”article, stated

“In our example, we have set the model’s temperature to 0 to ensure mostly deterministic outputs, as OpenAI’s models are non-deterministic by default.”article, stated

Models used: OpenAI completion model · LLMs

Scale

The workflow covers sentiment analysis, product aspect extraction, and topic clustering.
“For each task — whether sentiment analysis, product aspect extraction, or topic clustering — different models must be trained and maintained. But these models are often still considered “black boxes” as they lack explainability, further complicating the process.”article, stated
Temperature is set to 0 for mostly deterministic output.
“In our example, we have set the model’s temperature to 0 to ensure mostly deterministic outputs, as OpenAI’s models are non-deterministic by default.”article, stated

Alternatives considered (roads not taken)

Train separate traditional ML models per task.

Rejected because: That requires labeled data, multiple maintained models, and limited explainability.

“In a conventional Machine Learning workflow, extracting sentiment involves collecting a labeled dataset and training numerous models for different tasks.”article, stated

Use black-box sentiment classification only.

Rejected because: Stakeholders need highlighted evidence and aspect-level explanations.

“But these models are often still considered “black boxes” as they lack explainability, further complicating the process. Indeed, this methodology is not only intensive, but also consumes a great deal of time.”article, stated

Fine-tune a model for the review-analysis tasks.

Rejected because: Few-shot prompting can reduce the need for extensive fine-tuning.

“The advantages of few-shot learning include reducing the need for extensive fine-tuning — saving time and computational resources — and promoting generalization, enhancing the model’s adaptability to different tasks and scenarios.”article, stated

Patterns

`structured-output-contract` (analytics)

Prompts instruct the model to return properly formatted output for downstream processing.

“ensuring a properly formatted output for easier downstream processing”article, stated

`prompt-versioning` (prompt engineering)

The solution depends on robust, iterated prompt instructions rather than many trained models.

“Detailed model instructions are crucial for successful deployment”article, stated

`retrieval-ranking` (analytics)

Review aspects are grouped into broader topics for business interpretation.

“group these similar aspects under broader topics”article, stated

Tradeoffs

Modeling strategy

Chose Prompted LLM analytics over Separate conventional ML models.

Rationale: One promptable model can perform several analytics tasks with less development overhead.

“this methodology is not only intensive, but also consumes a great deal of time.”article, stated

Output behavior

Chose Temperature 0 over Higher-temperature creative generation.

Rationale: Analytics output needed consistency more than creativity.

“lower for deterministic responses”article, stated

Learning approach

Chose Few-shot prompts over Extensive fine-tuning.

Rationale: Few-shot examples reduce training cost and improve adaptability.

“providing the model with explicit instructions and a handful of examples”article, stated

Gotchas (what bit them)

LLM outputs still need determinism controls for analytics.

Root cause: OpenAI models are non-deterministic by default.

Resolution: Set temperature to 0 for mostly deterministic outputs.

“OpenAI’s models are non-deterministic by default.”article, stated

Traditional ML can be costly to maintain for multi-task analytics.

Root cause: Separate tasks require separate labels, models, and maintenance.

Resolution: Use a prompted LLM for multiple text-analytics tasks.

“different models must be trained and maintained”article, stated

Evaluation

The article evaluates through practical output qualities: deterministic behavior, formatted output, explainability, aspect extraction, sentiment assignment, and topic grouping for review analytics.

Metrics: Determinism · Formatted output · Explainability · Sentiment extraction · Aspect/topic grouping

“stakeholders see which sentiment was assigned”article, stated

Outcomes

The approach demonstrates sentiment, aspect, and topic analytics using one LLM workflow.
Stakeholders can inspect text spans that contributed to sentiment decisions.
Few-shot prompting reduces the need for extensive fine-tuning.

The source describes task breadth rather than production volume: sentiment analysis, aspect extraction, topic clustering, and temperature 0 deterministic output.

“LLMs can also pinpoint and highlight the specific sections in the review that contributed to this sentiment.”article, stated