Automated Inventory Counting with Multimodal LLMs in Grocery Fulfillment

picnic · E-commerce · 2025 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: E-commerce fulfillment

Picnic needed to automate stock counting in an automated grocery warehouse so inventory promises stayed accurate without repeatedly asking shoppers to manually count products in totes.

Constraints:

• totes move quickly through warehouse hardware with limited lighting
• inventory-count errors affect customer-facing stock availability
• some images are uncountable even for humans due to cardboard, stacking, or packaging
• model latency must be reasonable but does not need sub-second responses
• cost must support the business case for counting groceries at scale

Architecture

Picnic deploys 16 camera setups in its automated fulfillment center, each with camera, flashlight, laser, and barcode scanner to capture tote images and link them to expected products. A FastAPI service unlocks the camera immediately, then processes the image asynchronously with prompt template, tote image, and a supply-chain reference image. Requests flow through LiteLLM to Gemini, with a fine-tuned Gemini 2.5 Flash model chosen to reach Pro-level performance at Flash-tier economics. A rule-based accountability check filters uncountable images before counting, predictions go to warehouse control software and Snowflake, and a small percentage still receive manual shopper validation for continuous monitoring.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  camera-setup[Camera setup] --> fastapi-service[FastAPI service]
  fastapi-service --> context-assembler[Context assembler]
  context-assembler --> supply-chain-reference-images[Supply-chain reference images]
  context-assembler --> accountability-check[Accountability check]
  accountability-check --> litellm-proxy[LiteLLM proxy]
  litellm-proxy --> gemini-counting-model[Gemini counting model]
  gemini-counting-model --> warehouse-control-software[Warehouse control software]
  gemini-counting-model --> snowflake-monitoring[Snowflake monitoring]
  manual-validation-flow[Manual validation flow] --> snowflake-monitoring

Components

Data flow

A tote passes a camera setup, where the barcode scanner identifies the expected product and the camera captures the tote image. The FastAPI service acknowledges the camera immediately, then assembles the prompt, supply-chain reference image, and captured tote image. An accountability check determines whether the image is countable; countable images are sent through LiteLLM to Gemini for prediction. The prediction is sent to warehouse control software for operational decisions and to Snowflake for monitoring, while a small validation sample is also manually recounted by shoppers and published to Snowflake.

Why this architecture

Picnic's architecture is deliberately pragmatic. Weighing scales could not count individual items, and traditional CV models underperformed out of the box, so multimodal LLMs were selected for visual reasoning. But the team did not force LLMs everywhere: clustering failure cases showed a simple rule-based accountability check beat the model for countability. Supply-chain reference images supplied unique visual context that prompt optimization could not. Fine-tuning a Flash-tier model bridged the cost-performance gap, while LiteLLM preserved the ability to swap providers as model prices and quality changed.

Models used: Google Gemini 2.0 Flash · Google Gemini 2.5 Flash · Google Gemini 2.5 Pro · fine-tuned Gemini 2.5 Flash · multimodal LLMs

Scale

• Picnic operates in the Netherlands, Germany, and France.
  “with ideal you prepay your groceries in other countries that were active for instance in Germany and France often”transcript, implied
• The warehouse in Utrecht is described as one of Europe's most advanced automated warehouses.
  “So in Utreak we have actually one of the most advanced automated warehouses of Europe. um where uh in a normal normal”transcript, stated
• The dataset contained thousands of labeled pictures.
  “We have now a data set of thousands of pictures that are nicely labeled with a high quality uh confidence. We need some metrics uh”transcript, stated
• Picnic deployed 16 camera setups in the fulfillment center.
  “we bought 16 uh uh camera setups that are all spread now to FCA. We created a fast API web service that when it gets”transcript, stated
• Open-source vision models were estimated to trail proprietary models by about nine months on the chosen benchmark.
  “it's only 9 months approximately difference between having the best uh proprietary model being beaten by the best open source model”transcript, stated

Alternatives considered (roads not taken)

Weigh totes with installed scales

Rejected because: The scales had too much measurement deviation to infer individual product counts.

Traditional computer vision models

Rejected because: Out-of-box traditional CV models struggled with Picnic's specific grocery tote images despite the apparent simplicity of the task.

LLM-based accountability check

Rejected because: Prompting and prompt optimization performed poorly, while simple rules from clustered mistakes worked better.

Use Gemini 2.5 Pro for every count

Rejected because: It solved the problem but did not satisfy the cost side of the business case, so the team fine-tuned 2.5 Flash instead.

Patterns

fine-tuning-adaptation (model-optimization)

Picnic fine-tuned Gemini 2.5 Flash to outperform base 2.5 Pro while keeping Flash-tier economics.

model-routing (model-operations)

LiteLLM sits between Picnic's service and Gemini so the team can switch models as quality and prices change.

eval-driven-development (evaluation)

The team built labeled datasets, metrics, validation samples, and mistake analysis before choosing and operating the model.

observability-tracing (monitoring)

Predictions and sampled manual validation counts are published to Snowflake for continuous production monitoring.

Tradeoffs

Multimodal LLM over traditional CV

Chose Gemini multimodal models over YOLO/DETR/vision-transformer-style models out of the box.

Rationale: Traditional models offered lower latency and easier hosting, but Picnic's tested models performed poorly on the specific tote-counting problem.

Fine-tuned Flash model

Chose fine-tuned Gemini 2.5 Flash over base Gemini 2.5 Pro.

Rationale: The Pro model solved the task but was too costly for the business case; fine-tuning Flash reached sufficient performance at lower cost.

Model abstraction layer

Chose LiteLLM proxy in front of Gemini over hardcoding directly to one provider API.

Rationale: The team expected model quality and price to change quickly and wanted to capture improvements with minimal rework.

Gotchas (what bit them)

Some tote images are not countable even by humans.

Root cause: Cardboard, stacking, loose inner packaging, and occlusion can hide products from view.

Resolution: Picnic added an accountability check that routes uncountable cases away from model counting.

The LLM was the wrong tool for the countability precheck.

Root cause: The accountability failures clustered around simple visual/business-rule patterns.

Resolution: Embedding and clustering mistakes revealed groups that simple rules handled better than the model.

The counting model was distracted by warehouse-image artifacts.

Root cause: Stickers, reflections, upside-down products, and similar tote conditions confused the visual model.

Resolution: Adding supply-chain reference images as context dramatically improved accuracy and other metrics.

Evaluation

Picnic evaluates with exact count accuracy as the primary metric, MAPE for low-stock sensitivity, RMSE for larger errors, MSD for over/under-prediction bias, precision/recall for the accountability check, MMMU for model selection, and ongoing sampled manual recounts in production.

Metrics: exact count accuracy · mean average percentage error · root mean squared error · mean signed deviation · accountability precision and recall · manual validation sample in production

Outcomes

• Supply-chain reference images caused accuracy to improve sharply and other metrics to move in the right direction.
• Fine-tuned Gemini 2.5 Flash outperformed the base 2.5 Pro model for the business case.
• The system entered production with 16 camera setups, FastAPI, LiteLLM, Gemini, Snowflake monitoring, and sampled manual validation.

The deployment runs in a highly automated fulfillment center with 16 camera setups and uses sampled manual validation rather than full manual counting.