Building a Scalable Retriever-Ranker Architecture: Malt's Journey with Vector Databases and LLM-Powered Freelancer Matching

malt · Tech · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Freelancer-project matching and recommendation

Malt needed to replace a slow monolithic matching model with a scalable matching architecture that could search more than 700,000 freelancer profiles, keep recommendation quality high, support business filters such as location and availability, and prepare for more complex rankers.

Constraints:

• The original matching system could take up to one minute per query
• A cross-encoder over every project-freelancer pair was too slow at 700,000+ freelancers
• A bi-encoder alone was faster but less precise than a cross-encoder
• Filtering needed to include business constraints such as location and availability
• The system needed production availability, observability, and Kubernetes deployment

Architecture

Malt moved to a retriever-ranker architecture. The retriever encodes projects and precomputed freelancer embeddings with bi-encoder models, stores vectors in Qdrant, uses ANN search and business filters to quickly return a manageable candidate set, then sends that subset to the existing matching model acting as a ranker. Qdrant runs self-hosted on Kubernetes with clustering, sharding, replication, Prometheus/OpenMetrics, and Grafana monitoring.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  project-intake[Project Intake] --> project-encoder[Project Encoder]
  freelancer-profiles[Freelancer Profiles] --> freelancer-encoder[Freelancer Encoder]
  project-encoder[Project Encoder] --> qdrant[Qdrant]
  freelancer-encoder[Freelancer Encoder] --> qdrant[Qdrant]
  qdrant[Qdrant] --> retriever[Retriever]
  retriever[Retriever] --> ranker[Ranker]
  ranker[Ranker] --> recommendation-results[Recommendation Results]
  qdrant[Qdrant] --> kubernetes[Kubernetes]
  kubernetes[Kubernetes] --> monitoring[Monitoring]

Components

Data flow

Freelancer profiles are encoded ahead of time and stored as vectors. When a project is posted, Malt computes the project embedding in real time, queries Qdrant with ANN plus filtering to retrieve a relevant subset such as 1,000 candidate profiles, then ranks that subset with the current matching model and returns near-real-time recommendations.

Why this architecture

Malt chose a two-step retriever-ranker because cross-encoders were accurate but too slow across hundreds of thousands of profiles, while bi-encoders were fast but less precise. The vector database made ANN retrieval, filtering, scalability, and up-to-date vectors practical in production.

Models used: bi-encoder models · cross-encoder models · transformers · BERT · GPT

Scale

• The original matching model was released in 2021 and the new architecture work happened in 2023.
  “Back in 2021, we released the first version of our automatic matching between freelancers and projects, relying on a single, monolithic model. Over time, we brought more capabilities to this model, such as multilingual support, and improved its performance.”article, stated

  “In 2023, we made a significant leap forward thanks to a new architecture designed to boost the precision, speed, and scalability of our matching process.”article, stated
• The system had to search more than 700,000 freelancer profiles.
  “However, it is slow: for each project we receive (for instance, a client seeking a Java developer), we need to create a pairing with each freelancer (over 700,000 at the time of writing) and process each project-freelancer pair through the model, thus computing over 700,000 pairs!”article, stated
• The retriever example narrows 700,000+ freelancers to about 1,000 profiles.
  “The first step, the retriever, will quickly return a relevant subset, for example, 1,000 profiles with Java expertise out of our 700,000+ freelancers. However, this subset could include less relevant profiles than others (for instance, not the exact required version of Java or not within the”article, stated
• Qdrant benchmarked at 90 queries per second on the local benchmark.
  “|---------------|--------------------|---------------|----------------------| | Elasticsearch | 3 | 0.97 | Yes | | PGVector | 14 | 0.40 | Yes | | Qdrant | 90 | 0.84 | Yes | | Pinecone | / | / | No | About this benchmark:”article, stated
• The benchmark dataset had 1,000,000 vectors with 960 dimensions.
  “* Dataset used is GIST1M Texmex corpus. It includes 1 000 000 vectors, each with 960 dimensions * Queries per second were measured on a local laptop (Macbook pro with M1 pro chip) using l2 as the similarity measure and 100 candidates requested”article, stated
• Production p95 latency dropped to at most 3 seconds.
  “milestone for our matching system. We saw a huge improvement in response times, with p95 latency dropping from tens of seconds (sometimes taking over a minute!) to a snappy 3 seconds at most. All this was achieved without sacrificing the quality of our recommendations, which was confirmed”article, stated

Alternatives considered (roads not taken)

Use a cross-encoder for all pairs

Rejected because: It was highly precise but required scoring every project against 700,000+ freelancers.

Use only a bi-encoder

Rejected because: Bi-encoders are fast but lose some pairwise relationship quality, so Malt retained a ranking stage for precision.

Use Elasticsearch or PGVector

Rejected because: Elasticsearch underperformed for high-dimensional vector search, and PGVector missed speed and precision benchmarks.

Patterns

retrieval-ranking (retrieval)

Malt splits matching into recall-oriented retrieval and precision-oriented ranking.

latency-optimization (performance)

The retriever reduces the candidate pool before ranking, cutting p95 latency to at most 3 seconds.

eval-driven-development (evaluation)

Malt benchmarked vector databases and validated production quality through AB testing.

hybrid-search (future-direction)

The team calls out hybrid semantic and keyword search as an important future feature for cases where keyword search is better.

Tradeoffs

Use retriever-ranker instead of one model

Chose Bi-encoder retriever plus ranker over A single monolithic matching model.

Rationale: The two-stage design balances recall, precision, latency, and future extensibility.

Self-host Qdrant on Kubernetes

Chose Self-hosted Qdrant with Helm on Kubernetes over Familiar databases or managed alternatives that lacked fit.

Rationale: Qdrant met speed, ANN, filtering, and geo-spatial requirements while Kubernetes gave reproducible deployment.

Deploy retriever first before the next-generation ranker

Chose Retriever before the existing matching model over Waiting for the full new retriever-ranker stack.

Rationale: This let Malt validate scalability and performance while the more complex ranker remained in development.

Gotchas (what bit them)

Filtering outside the vector database can damage performance or quality.

Root cause: Pre-filtering creates huge masks and can degrade ANN graph quality; post-filtering requires requesting many extra candidates.

Resolution: Malt selected a vector database with strong filtering, especially geo-spatial filtering.

Cross-encoder quality did not make it production-scalable.

Root cause: The model would need to score each project against every freelancer.

Resolution: Use a fast retriever first, then a ranker on a narrowed subset.

Evaluation

Malt evaluated database candidates on query speed, ANN quality, and filtering under use-case-like conditions, then evaluated production impact with p95 latency and AB-test conversion.

Metrics: queries per second · ANN precision · filtering support · p95 latency · AB-test project conversion

Outcomes

• Malt deployed the retriever-ranker architecture in 2023
• p95 latency dropped from tens of seconds or over a minute to at most 3 seconds
• Recommendation quality was maintained, with AB testing showing increased project conversion

The architecture handles 700,000+ freelancer profiles and retrieves approximately 1,000 candidates for ranking in the example.