Scaling Product Categorization with Batch Inference and Prompt Engineering

godaddy · E-commerce · 2025 · source Agentic: no · promotional bias: high · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: ecommerce product category generation

GoDaddy needed to categorize millions of products from product names with higher coverage and lower cost than an existing per-product Llama 2 approach that produced incomplete or mislabeled categories.

Constraints:

• The catalog contains six million products.
• The solution needed high coverage for downstream predictive modeling.
• Individual LLM calls were too costly and slow at this scale.

Architecture

GoDaddy and AWS built a Bedrock batch-inference pipeline. Product JSONL files uploaded to S3 trigger Lambda; Bedrock batch processing reads prompts, generates categorized outputs, writes JSONL output to S3; a second Lambda monitors the job and shuts down the endpoint; LangChain PydanticOutputParser and OutputFixingParser turn model JSON into CSV. N-packing shares prompt instructions across multiple SKUs, and model/prompt comparisons selected Claude-Instant zero-shot for latency, cost, and coverage.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  A["S3 input bucket"] --> B["Start Lambda"]
  B --> C["Bedrock batch job"]
  D["Claude and Llama models"] --> C
  C --> E["S3 output bucket"]
  C --> F["Monitor Lambda"]
  E --> G["Parsing layer"]
  G --> H["Evaluation framework"]

Components

Data flow

A JSONL product file lands in S3, Lambda starts a Bedrock batch job, Bedrock reads product names and prompts, writes categorized JSONL output back to S3, a second Lambda monitors/cleans up the job, and the output file is parsed with PydanticOutputParser and OutputFixingParser into structured category data.

Why this architecture

Batch inference and n-packing reduce cost/latency by sharing prompt instructions across SKUs and avoiding per-product calls, while structured parsing and model comparison improved coverage and reliability.

Models used: Llama2-13b · Llama2-70b · Claude-Instant · Claude-v2

Scale

• The starting catalog had six million products.
  “generate the product categories for six million products where a product is identified by an SKU.”article, stated
• The evaluation used 30 labeled points and 100,000 unlabeled test points.
  “We used a dataset that consisted of 30 labeled data points and 100,000 unlabeled test data points.”article, stated
• Scaling from 5,000 to 100,000 samples required only eight times more computation time.
  “When scaling from 5,000 to 100,000 samples, only eight times more computation time was needed.”article, stated
• The recommended solution processed 5,000 products in 12 minutes.
  “Our solution inferences 5,000 products in 12 minutes, which is 80% faster than GoDaddy’s needs”article, stated

Alternatives considered (roads not taken)

Out-of-the-box Llama 2 per-product categorization

Rejected because: It produced incomplete/mislabeled categories and was costly when called individually.

Individual LLM calls for each product

Rejected because: It would increase inference time far more than batch processing at 100,000 products.

Use the same LLM for both generation and parsing in all cases

Rejected because: The team observed Llama 2 was weaker at format parsing and Claude was more suitable for parsing.

Patterns

batch-offline-processing (cost-optimization)

Amazon Bedrock batch jobs process large JSONL product categorization workloads asynchronously.

structured-output-contract (output-control)

A Pydantic schema and parsers enforce structured category fields from LLM outputs.

model-routing (model-operations)

The team compared Llama and Claude variants and identified where each model family performed better.

eval-driven-development (evaluation)

Coverage, parsing, exact-match, and human SME evaluation guided model/prompt selection.

Tradeoffs

Inference mode

Chose Bedrock batch inference over per-product online calls.

Rationale: Batch scaling made 100,000-product categorization far more time-efficient and cost-effective.

Packing strategy

Chose n-packing multiple SKUs per query over one product per prompt.

Rationale: Shared prompt instructions reduce latency and cost.

Prompt examples

Chose limit large-scale testing to at most 5-shot over 10-shot everywhere.

Rationale: More examples may improve performance but increase cost around 30%.

Gotchas (what bit them)

JSON formatting is a production failure mode, not a cosmetic issue.

Root cause: Models can produce malformed or hard-to-parse outputs, especially weaker format-following models.

Resolution: Use PydanticOutputParser, OutputFixingParser, and prompt-engineered JSON instructions.

Packing changes model behavior.

Root cause: Some models generalize poorly as packing number varies across 1, 5, and 10.

Resolution: Evaluate packing per model family and prefer Claude for stable generalization.

Ground truth with missing values limits downstream utility.

Root cause: Some labeled fields contained N/A or missing values, but GoDaddy wanted high coverage.

Resolution: Optimize final coverage and recommend larger/cleaner ground-truth datasets.

Evaluation

GoDaddy evaluated content coverage, parsing coverage, parsing recall/precision on product name, final coverage, human SME quality, latency, cost, and scaling from 5k to 100k samples.

Metrics: content coverage · parsing coverage · parsing recall · parsing precision · final coverage · latency · cost · human SME quality

Outcomes

• Claude-Instant zero-shot achieved 97% category coverage on 5,000 and 100,000 hold-out sets.
• The solution processed 5,000 products in 12 minutes, 80% faster than the one-hour requirement.
• The recommended Claude-Instant solution was 8% more affordable than the Llama2-13B proposal and provided 79% more coverage.
• The framework supports both batch and near-real-time inference around 2 seconds per product.

The source is an AWS customer story and should be read as a vendor-supported case study, but it provides concrete latency, coverage, and cost metrics.