Fine-Tuning and Quantizing LLMs for Dynamic Attribute Extraction

mercari · E-commerce · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: e-commerce marketplace attribute extraction

Mercari needed to extract dynamically specified product attributes from user-generated marketplace listings whose content, categories, languages, and attribute requirements vary over time, while avoiding the cost and hallucination-control limits of commercial LLM APIs.

Constraints:

seller descriptions vary widely
important attributes differ by category
user-generated content evolves over time
commercial API cost was prohibitive for production request volumes
prompt-only control over hallucinations was weaker than owning a fine-tuned model

Architecture

Mercari built a task-specific extraction model by defining listing-description plus attribute-key inputs, preparing historical listing/attribute data, selecting Google gemma-2b-it after model evaluation, fine-tuning it with QLoRA on a single A100 GPU, converting the fine-tuned Hugging Face model to GGUF with llama.cpp, applying 4-bit post-training quantization, and evaluating quality, size, latency, and cost against GPT-3.5 Turbo.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  listing-and-attribute-input[Listing and attribute input]
  historical-attribute-dataset[Historical attribute dataset]
  qlora-fine-tuned-gemma-model[QLoRA fine-tuned Gemma model]
  llama-cpp-conversion-and-quantization[llama.cpp conversion and quantization]
  evaluation-harness[Evaluation harness]
  listing-and-attribute-input --> qlora-fine-tuned-gemma-model
  historical-attribute-dataset --> qlora-fine-tuned-gemma-model
  qlora-fine-tuned-gemma-model --> llama-cpp-conversion-and-quantization
  llama-cpp-conversion-and-quantization --> evaluation-harness
  evaluation-harness --> qlora-fine-tuned-gemma-model

Drag to pan · scroll or use +/− to zoom · reconstructed from the source's described component topology

Components

Component	Role	Interacts with
`Listing and attribute input`	Marketplace listing description plus dynamically requested attribute keys.	`QLoRA fine-tuned Gemma model`
`Historical attribute dataset`	Prepared examples of descriptions and attributes used for fine-tuning.	`QLoRA fine-tuned Gemma model`
`QLoRA fine-tuned Gemma model`	Task-specialized 2B model trained to extract requested attributes from listing text.	`llama.cpp conversion and quantization`
`llama.cpp conversion and quantization`	Converts the fine-tuned model to GGUF and applies 4-bit quantization for smaller/faster serving.	`Evaluation harness`
`Evaluation harness`	Compares extracted values against references and commercial API baselines using BLEU, model size, latency, and cost.	`QLoRA fine-tuned Gemma model`

Data flow

The extraction task takes a listing description and a list of requested attribute keys, produces key-value outputs, trains gemma-2b-it with QLoRA on prepared marketplace examples, converts the fine-tuned model to llama.cpp-compatible GGUF, quantizes it to 4-bit precision, and evaluates the result against GPT-3.5 Turbo on BLEU, model size, latency, and cost readiness.

"### 入力（文章）: {input}

### 応答:
{output}"

“Convert the fine-tuned model, previously stored in the HuggingFace format, to a format compatible with llama.cpp.”article, stated

Why this architecture

Mercari chose a small fine-tuned model because dynamic category-specific extraction required better control and lower production cost than commercial APIs, while QLoRA and quantization offered a practical path to task specialization without large-model infrastructure.

“Cost of commercial APIs: Though commercial LLM APIs are becoming more affordable, at the time this article is written, the sheer number of requests in a production environment would still make them prohibitively expensive.”article, stated

“Control over hallucinations: It’s more difficult to manage and minimize hallucinations purely through prompt engineering with commercial APIs.”article, stated

Models used: gemma-2b-it · GPT-3.5 Turbo

Scale

The experiment fine-tuned a 2-billion-parameter model.
“This article will describe one of our past experiments in fine-tuning a 2-billion parameter large language model (LLM) using QLoRA”article, stated
Fine-tuning used a single A100 GPU with 80 GB memory.
“we used a single A100 GPU with an 80 GB memory VM instance (a2-ultragpu-1g) from GCP to fine-tune a large language model using QLoRA.”article, stated
The final 4-bit model was approximately 95% smaller than the base gemma-2b-it model.
“The model is approximately 95% smaller than the gemma-2b-it base model downloaded from HuggingFace.”article, stated
The fine-tuned model was estimated to reduce cost by more than 14 times compared with GPT-3.5 Turbo.
“using the fine-tuned model could reduce the cost by more than 14 times compared to using gpt-3.5-turbo-0125.”article, stated

Alternatives considered (roads not taken)

lightweight conventional non-LLM models

Rejected because: dynamic attributes, category-specific requirements, and multilingual listings made conventional models high-maintenance and less generalizable.

“Dynamic and varied attributes: The way attributes are described can change frequently, leading to high maintenance requirements and the need for continuous model re-training.”article, stated

“Multi-linguality: Most listings in Mercari are written in Japanese, however, with the huge variety of goods being exchanged, there are also listings written in other languages”article, stated

commercial LLM APIs

Rejected because: the request volume made API cost prohibitive and hallucination control harder through prompting alone.

“the sheer number of requests in a production environment would still make them prohibitively expensive.”article, stated

“It’s more difficult to manage and minimize hallucinations purely through prompt engineering with commercial APIs.”article, stated

Patterns

`fine-tuning-adaptation` (model-operations)

Mercari fine-tuned gemma-2b-it with QLoRA for the specific task of extracting dynamically specified attributes from marketplace listing descriptions.

“fine-tuning a 2-billion parameter large language model (LLM) using QLoRA, to extract dynamically specified attributes from user-generated content”article, stated

`latency-optimization` (serving)

The team used llama.cpp conversion and 4-bit quantization to reduce model size and support faster inference.

“Post-training quantization aims to see if we can further shrink the model size while maintaining satisfactory performance.”article, stated

“The 4-bit precision method (q4_k_m) worked well for our use case.”article, stated

`eval-driven-development` (evaluation)

Mercari compared the quantized model with GPT-3.5 Turbo using BLEU score, model size, and latency/cost readiness metrics.

“BLEU Score: This score provided insights into the quality of extracted attribute values compared to the actual values.”article, stated

“Model Size and Latency: We also checked the resulting model size and latency to assess cost-efficiency and readiness for production use.”article, stated

Tradeoffs

Fine-tune a small open model for attribute extraction.

Chose QLoRA-tuned gemma-2b-it over GPT-3.5 Turbo API for the production path.

Rationale: The small fine-tuned model outperformed the larger API baseline on extraction quality while being smaller and cheaper.

“Results show that the fine-tuned model outperforms the bigger model in terms of extraction quality while being significantly smaller in size and less costly.”article, stated

Quantize after fine-tuning.

Chose 4-bit GGUF model via llama.cpp over serving the larger fine-tuned Hugging Face model unchanged.

Rationale: Quantization was used to shrink the model while maintaining satisfactory performance for serving readiness.

“Post-training quantization aims to see if we can further shrink the model size while maintaining satisfactory performance.”article, stated

Gotchas (what bit them)

Attribute extraction requirements change across categories and over time.

Root cause: Seller descriptions are varied, essential attributes differ by category, and user-generated content evolves continuously.

Resolution: Mercari used an LLM fine-tuned for dynamically specified attribute keys instead of maintaining many brittle conventional models.

“Wide variety of user-generated content: Each seller describes their listings differently.”article, stated

“Category specificity: What’s essential varies from one category to another.”article, stated

Commercial LLM APIs were hard to control and expensive at production request volume.

Root cause: High request volume drove API cost, while prompt engineering alone provided weaker hallucination control.

Resolution: Mercari fine-tuned and quantized its own smaller model for the extraction task.

“the sheer number of requests in a production environment would still make them prohibitively expensive.”article, stated

“It’s more difficult to manage and minimize hallucinations purely through prompt engineering with commercial APIs.”article, stated

Evaluation

Mercari evaluated the quantized model against GPT-3.5 Turbo using BLEU score for extracted values plus model size and latency/cost readiness metrics.

Metrics: BLEU score · model size · latency · cost-efficiency · comparison to GPT-3.5 Turbo

“BLEU Score: This score provided insights into the quality of extracted attribute values compared to the actual values.”article, stated

“Model Size and Latency: We also checked the resulting model size and latency to assess cost-efficiency and readiness for production use.”article, stated

Outcomes

the 4-bit quantized fine-tuned model was approximately 95% smaller than the base model
the model achieved a BLEU score more than five percentage points above GPT-3.5 Turbo
the estimated cost reduction was more than 14x compared with GPT-3.5 Turbo
fine-tuning improved control over hallucinations compared with prompt-only commercial API use

The source describes an experiment rather than a full production launch, but the motivation was production request volume and production cost/latency readiness.

“The model achieved a BLEU score slightly more than five percentage points higher than gpt-3.5-turbo-0125.”article, stated

“using the fine-tuned model could reduce the cost by more than 14 times compared to using gpt-3.5-turbo-0125.”article, stated

Fine-Tuning and Quantizing LLMs for Dynamic Attribute Extraction

Problem

Architecture

Components

Data flow

Why this architecture

Scale

Alternatives considered (roads not taken)

lightweight conventional non-LLM models

commercial LLM APIs

Patterns

fine-tuning-adaptation (model-operations)

latency-optimization (serving)

eval-driven-development (evaluation)

Tradeoffs

Fine-tune a small open model for attribute extraction.

Quantize after fine-tuning.

Gotchas (what bit them)

Attribute extraction requirements change across categories and over time.

Commercial LLM APIs were hard to control and expensive at production request volume.

Evaluation

Outcomes

`fine-tuning-adaptation` (model-operations)

`latency-optimization` (serving)

`eval-driven-development` (evaluation)