Automating Job Role Extraction Using Prosus AI Assistant in Production

olx · E-commerce · 2024 · source Agentic: no · promotional bias: low · summary-only (no fetched source) · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Marketplace job-search enrichment

OLX needed to extract normalized job roles from unstructured job-ad titles and descriptions so job seekers could find relevant listings more effectively.

Constraints:

• job roles were buried in ad text rather than clean taxonomies
• production traffic includes about 2,000 new or updated ads per day
• external LLM API calls create cost and latency considerations
• search impact must be validated with A/B testing
• category and taxonomy evolution must be handled over time

Architecture

OLX built a multi-stage job-role extraction pipeline using Prosus AI Assistant, an OpenAI-backed internal service. The system preprocesses ads, trims descriptions for token/cost control, translates where needed, extracts job roles, matches them into standardized taxonomies, publishes extracted roles to AWS Kinesis, and feeds the search indexing pipeline. A separate taxonomy-generation workflow uses searched keywords and extracted roles to build hierarchical job-role trees.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  ad-event-subscriber[Ad event subscriber] --> preprocessing-layer[Preprocessing layer]
  preprocessing-layer --> prosus-ai-assistant[Prosus AI Assistant]
  prosus-ai-assistant --> aws-kinesis-stream[AWS Kinesis stream]
  aws-kinesis-stream --> search-indexing-pipeline[Search indexing pipeline]
  taxonomy-generator[Taxonomy generator] --> search-indexing-pipeline
  prosus-ai-assistant --> taxonomy-generator

Components

Data flow

New or updated ads are cleaned and trimmed, sent to Prosus AI Assistant for job-role extraction and taxonomy matching, emitted through AWS Kinesis, and incorporated into the search index. Periodically, searched keywords and sampled job roles feed taxonomy-tree generation when category structures change.

Why this architecture

OLX decomposed the job into extraction, taxonomy generation, and indexing because role discovery and search retrieval have different cadences. Continuous extraction handles daily ad updates, while taxonomy regeneration happens only when category trees change. Using Prosus AI Assistant accelerated deployment under privacy terms, but the cost profile pushed the team toward evaluating self-hosted models.

Models used: Prosus AI Assistant · OpenAI-backed LLM service

Scale

• The production pipeline handles about 2,000 new or updated ads per day.
  “the system handles approximately 2,000 newly created or updated ads daily.”summary, stated
• The split extraction/matching design produces about 4,000 API requests daily.
  “The team made an architectural decision to break down the processing into two sub-tasks—job-role extraction and matching within the standardized tree—resulting in approximately 4,000 daily API requests to Prosus AI Assistant.”summary, stated
• The proof of concept sampled 2,000 job ads.
  “They sampled 2,000 job ads for their proof of concept, accounting for uneven distribution across sub-categories to ensure representative coverage.”summary, stated
• Job-role taxonomy prompts used up to 100 searched keywords and up to 50 extracted job roles.
  “This involved providing the LLM with up to 100 profession-related searched keywords and up to 50 job roles extracted from randomly selected job ads within each category.”summary, stated
• The service cost about $15,000 per month.
  “The case study provides valuable transparency about operational costs: approximately $15,000 per month for the Prosus AI Assistant service.”summary, stated

Alternatives considered (roads not taken)

Self-hosted LLMs

Rejected because: Prosus AI Assistant enabled faster deployment and privacy handling, though self-hosting remained a future cost-control option.

Single monolithic model task

Rejected because: The team split processing into extraction and taxonomy matching, which doubled request count but simplified the task.

Integrate extracted roles directly into ranking first

Rejected because: The initial A/B test used retrieval only, leaving ranking integration for later.

Patterns

structured-output-contract (data)

The LLM extracts unstructured job ads into normalized job-role taxonomy data for indexing.

prompt-chaining (workflow)

The production path decomposes extraction and matching into two LLM sub-tasks.

cost-aware-routing (operations)

OLX trims inputs, separates periodic taxonomy generation from daily extraction, and evaluates self-hosting because API costs are significant.

eval-driven-development (evaluation)

OLX used A/B testing and segmented search metrics to validate search impact.

Tradeoffs

Use Prosus AI Assistant first

Chose external/internal managed LLM API over self-hosted models at launch.

Rationale: The API offered accuracy, privacy terms, and faster time-to-market, but created long-term cost and dependency concerns.

Split extraction and taxonomy matching

Chose two sub-tasks over one LLM call per ad.

Rationale: The split improved task clarity but doubled daily API request volume.

Trim ad descriptions before model calls

Chose first 200 words/tokens over full ad text.

Rationale: Input trimming managed API costs and token limits while preserving enough context for role extraction.

Gotchas (what bit them)

External API cost became a strategic concern.

Root cause: Daily extraction required around 4,000 API requests and cost about $15,000 per month.

Resolution: The team began evaluating self-hosted models for long-term sustainability.

Search impact was limited by retrieval-only integration.

Root cause: Extracted roles were not yet integrated into ranking.

Resolution: The team treated initial A/B results as promising but limited.

Taxonomy evolution can create inconsistencies.

Root cause: Changing job categories may require recreating role taxonomies.

Resolution: OLX planned automated sub-category change detection and taxonomy regeneration.

Evaluation

OLX used A/B testing focused on retrieval-stage search metrics and segmented results by low/medium/high result-count groups.

Metrics: Successful Events uplift · search extensions · keyword searches per user · confidence intervals · retrieval-stage impact

Outcomes

• The system processes about 2,000 new or updated ads daily.
• The two-step design generates about 4,000 daily API requests.
• A/B testing showed positive uplift in Successful Events metrics.
• Search extensions and keyword searches decreased when results were fewer than 50.
• The $15K monthly cost prompted evaluation of self-hosted models.

Summary-only evidence reports daily ad/API volume, $15K monthly cost, and retrieval-stage search improvements.