Evaluating Product Image Integrity in AI-Generated Advertising Content

microsoft · Media & Entertainment · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: quality assurance for AI-generated advertising images

Microsoft needed a measurable way to verify that AI-generated ad backgrounds did not alter the original product image in personalized advertising outputs.

Constraints:

The ad workflow uses inpainting, which can slightly modify the product near the generated mask.
Product representation accuracy matters for advertising integrity.
A single image-comparison metric cannot detect all changes across color, scale, translation, rotation, edges, and curves.
Product location may change inside the generated output.
Template matching can miss additions outside the detected product bounding box.

Architecture

Microsoft's ISE team combined multiple image-comparison techniques. The pipeline compares the baseline product image against the product area detected inside the generated ad image. Template matching locates the product region, MSE and PSNR measure pixel/color and scale differences, and cosine similarity over CNN feature vectors captures edges and curves. Benchmark images with known changes are used to understand each method's strengths and failure modes.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  product-image-input["Product image input"] --> inpainting-generator["Inpainting generator"]
  mask-and-text-prompt["Mask and text prompt"] --> inpainting-generator
  inpainting-generator --> template-matching["Template matching"]
  template-matching --> mse-and-psnr-comparators["MSE and PSNR comparators"]
  template-matching --> cosine-similarity-comparator["Cosine similarity comparator"]
  mse-and-psnr-comparators --> integrity-decision["Integrity decision"]
  cosine-similarity-comparator --> integrity-decision

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Product image input`	Original product image that must remain unchanged.	`Inpainting generator`
`Mask and text prompt`	Defines the generated background area and requested surroundings.	`Inpainting generator`
`Inpainting generator`	Generates the ad background around the product image.	`Template matching`
`Template matching`	Locates the product region inside the generated image.	`MSE and PSNR comparators`, `Cosine similarity comparator`
`MSE and PSNR comparators`	Detect pixel color and disproportionate scale differences.	`Integrity decision`
`Cosine similarity comparator`	Compares CNN feature vectors for edges and curves.	`Integrity decision`
`Integrity decision`	Combines comparator signals to decide whether the product was altered.	—

Data flow

The system starts with a product image, mask, and prompt for inpainting. After the model generates the ad background, template matching finds the product in the generated image, scopes the relevant region, and applies MSE, PSNR, and cosine similarity to compare product integrity against the original.

“With inpainting, the inputs are:”article, stated

“Template matching serves to provide a baseline of which to compare the images. By finding where the product is in the generated image”article, stated

Why this architecture

Microsoft combined comparators because each method catches different product-integrity failures. MSE and PSNR are strong for pixel color and disproportionate scale when aligned, cosine similarity captures edges and curves even under translation or proportional scale, and template matching scopes comparisons to the located product region.

“The strength of combining multiple image comparators is that we can leverage the benefits of each, and they complement each other’s shortcomings.”article, stated

“This helps when running MSE and PSNR due to the nature of how those comparison techniques are performed and gives us a more meaningful result”article, stated

Models used: Multi-modal text-to-image inpainting model · Mean Squared Error · Peak Signal-to-Noise Ratio · CNN feature vectors · Cosine Similarity · OpenCV template matching

Scale

The customer goal was 1:1 ad personalization.
“They were aiming for 1:1 ad personalization, where each content piece is tailored to each unique customer”article, stated
The method used benchmark images with known deltas.
“a series of benchmark images were used. These are images where the delta between the images is known”article, stated

Alternatives considered (roads not taken)

Use only MSE or PSNR.

Rejected because: Pixel metrics fail when the product is translated and cannot discern acceptable rotation or proportional scale.

“MSE isn’t useful when the product was translated unless a template matcher is used.”article, stated

Use only cosine similarity.

Rejected because: Cosine similarity misses pixel color changes and undesirable disproportionate scaling.

“It also won’t detect pixel color changes as well as MSE or PSNR due to it only comparing features.”article, stated

Trust inpainting to preserve the product.

Rejected because: The generated output may slightly modify product content near the masked background.

“Since the output of inpainting cannot guarantee that the product was not modified, we needed a way to measure if the generated images altered the product.”article, stated

Patterns

`eval-driven-development` (image integrity)

Microsoft built benchmark images and comparator measurements to evaluate product modification in generated ads.

“To test the capabilities of each image comparator technique, a series of benchmark images were used.”article, stated

`guardrails-io-filtering` (output quality control)

The image comparison system gates AI-generated ad outputs on whether the original product appears altered.

“we can achieve a strong system of image comparison to determine whether the product has been edited in the AI-generated image output.”article, stated

`structured-output-contract` (measurement contract)

The pipeline decomposes product-integrity measurement into explicit comparator outputs rather than relying on visual inspection alone.

“Three measurements techniques were used in determining the delta between the base product image and the generated ad image:”article, stated

Tradeoffs

Combine pixel and feature comparators.

Chose MSE/PSNR plus cosine similarity over A single comparator.

Rationale: Pixel metrics and feature similarity catch different failure modes.

“With MSE and PSNR, the key benefit is in determining color differences and disproportionate scaling. With Cosine Similarity, the key benefit is determining differences in the edges and curves”article, stated

Use template matching to scope pixel comparisons.

Chose OpenCV product localization over Comparing whole generated and original images directly.

Rationale: The product may move within the generated image, so the comparison needs to locate the product first.

“This was done using template matching, which is a method used to locate a smaller image (the ‘template’) within a larger image.”article, stated

Gotchas (what bit them)

Inpainting can alter product content.

Root cause: The generated background may touch or modify product pixels near masked areas.

Resolution: Measure product deltas after generation with image-comparison techniques.

“The product content may have been slightly modified to fit into the background, especially if the masked areas are close to the product content.”article, stated

Template matching can miss changes outside the product bounding box.

Root cause: Scoping comparison only to the detected product region ignores inaccurate additions adjacent to the product.

Resolution: Further research is needed to detect additions outside the bounding box.

“detecting additions to the image outside of the bounding box of the template.”article, stated

Resolution mismatches complicate template matching.

Root cause: OpenCV does not track resolution, and generated images may use model-specific resolutions.

Resolution: Keep template and target resolutions the same or use resizing techniques.

“We recommend keeping the resolution the same between the template and the target image.”article, stated

Evaluation

Microsoft tested MSE, PSNR, cosine similarity, and template matching against benchmark images with known deltas, including fill, transparency, rotation, translation, coloring, and scaling changes.

Metrics: Mean Squared Error · Peak Signal-to-Noise Ratio · Cosine similarity · Template match location · Known image delta detection · Color difference · Scale difference · Edge and curve similarity

“The images include tests such as various levels of image fill, transparency fill, content rotation, translation, coloring, and image scaling.”article, stated

Outcomes

Microsoft identified a combined comparator system for detecting whether generated ads altered the product.
MSE and PSNR worked well for product color and disproportionate scale differences when location was controlled.
Cosine similarity worked better for edges and curves regardless of location or proportional scale.
Template matching scoped comparisons to the product region but left open issues around rotation, scaling, and additions outside the bounding box.

The case does not report production traffic, but it targets 1:1 personalization and uses known-delta benchmark images to validate comparator behavior.

“A combination of MSE (or PSNR) and Cosine Similarity comparators can give us meaningful metrics to determine undesired changes”article, stated

“Further considerations include handling scaled and rotated images as part of the template matching”article, stated