Building a Modern Search Engine for Parliamentary Records with RAG Capabilities

hansard · Government · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.4 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Public-sector search and RAG retrieval

Singapore's Pair Search team needed to make decades of Hansard parliamentary records searchable for policy officers, civic users, and downstream RAG systems because the existing keyword-only search produced poor rankings and weak snippets.

Constraints:

• Hansard records span changing formats from 1955 onward
• the legacy search ranked frequent keyword mentions above phrase intent
• users needed useful snippets to judge whether a result was relevant
• the system needed state-of-the-art retrieval without hurting latency
• the search engine should serve humans directly and act as a retrieval stack for government LLM products

Architecture

Pair Search uses Vespa as a scalable serving engine over a standardized Hansard corpus. The pipeline has three phases: document processing for scraping, parsing, and normalizing Hansard records; retrieval using keyword matching with weakAnd/nativeRank/BM25 plus semantic e5 embeddings; and reranking with ColBERTv2 plus a global hybrid score that combines semantic, keyword, and reranker signals. The system also exposes base search and RAG retrieval APIs for other applications.

Diagram confidence: explicit (topology described by the source).

flowchart LR
  hansard-corpus[Hansard corpus] --> document-processing[Document processing]
  document-processing --> vespa-search-engine[Vespa search engine]
  vespa-search-engine --> keyword-retrieval[Keyword retrieval]
  vespa-search-engine --> semantic-retrieval[Semantic retrieval]
  vespa-search-engine --> colbertv2-reranker[ColBERTv2 reranker]
  keyword-retrieval --> global-hybrid-scorer[Global hybrid scorer]
  semantic-retrieval --> global-hybrid-scorer
  colbertv2-reranker --> global-hybrid-scorer
  global-hybrid-scorer --> search-ui-and-api[Search UI and API]
  search-ui-and-api --> pair-chat-assistants[Pair Chat assistants]

Components

Data flow

Hansard reports are scraped, parsed, and standardized into a uniform search format. At query time, Vespa retrieves candidates with keyword-based weakAnd/nativeRank/BM25 and semantic e5 embeddings. Content-node reranking narrows the set, ColBERTv2 reranks more deeply, and a global phase merges top results from all content nodes using a hybrid score. The same search index can support direct user search, snippets, RAG retrieval, and future LLM-assisted query expansion or index enrichment.

4 supporting quotes

“ The search process in Pair Search is meticulously divided into three phases: 1. Document-Processing:”article, stated

“The keyword-based search utilizes Vespa's weakAnd operator alongside nativeRank and BM25 text matching algorithms to efficiently sift through vast amounts of data.”article, stated

“Pair Search incorporates e5 embeddings, which offer a balance of speed, cost-effectiveness, and enhanced performance compared to alternatives like OpenAI's ada embeddings model.”article, stated

“Phase 3 (global phase): The final phase amalgamates the top results from all content nodes, creating a hybrid score that integrates semantic, keyword-based, and ColbertV2 scores.”article, stated

Why this architecture

Pair Search chose a hybrid retrieval and reranking architecture because keyword-only search over parliamentary records overweights frequent terms, while semantic-only or single-score approaches can bias toward one dimension of quality. A staged pipeline keeps latency acceptable while allowing more expensive reranking only after cheaper narrowing steps.

Models used: e5 embeddings · ColBERTv2 · Vespa weakAnd · nativeRank · BM25

Scale

• Hansard records go back to 1955.
  “These reports can be found as far back as 1955, when Parliament was known as the Legislative Assembly. This treasure trove of information is used by policy makers, professionals and members of the public alike.”article, stated
• Pair Search indexed the full Hansard database from 1955 through January 2024.
  “The site is currently live, and indexed on the full Hansard Database from 1955 - January 2024, spanning over 30,000 reports. Already we are seeing dramatic improvements in search results using Pair Search.”article, stated
• The Hansard database contains over 30,000 reports.
  “This site contains over 30,000 reports starting from 1955. Given the evolving data formats over the decades, standardizing this diverse information for search purposes presented significant hurdles.”article, stated
• Early soft launch usage averaged about 150 daily users and 200 daily searches.
  “users who frequently utilize Hansard Search, including Ministry of Law legal policy officers, Comm Ops officers at MCI and PMO, COS coordinators among others. We are averaging ~150 daily users and ~200 daily searches in the early days of our soft launch, and initial user feedback has been”article, stated
• The prototype was built during Hack for Public Good 2024.
  “Pair Search was prototyped during OGP’s Hack for Public Good 2024 Hackathon, under the guiding principles of proving search that should be: * State of the Art (SOTA) - Search should leverage on all the recent advances in search technology”article, stated

Alternatives considered (roads not taken)

Legacy 100% keyword search

Rejected because: Frequent single-word matches crowded out documents matching the phrase intent, and results lacked helpful snippets.

OpenAI ada embeddings

Rejected because: The team selected e5 embeddings as a better balance of speed, cost, and performance.

Rely on one ranking signal

Rejected because: Single metrics over-bias toward one dimension of relevance, so Pair Search uses a hybrid score.

Launch with Magic Summary first

Rejected because: The team rolled out without generated summaries to focus first on the core search experience.

Patterns

hybrid-search (knowledge_retrieval)

Pair Search combines keyword-based retrieval with semantic e5 embeddings for Hansard queries.

retrieval-ranking (knowledge_retrieval)

Pair Search uses cost-effective narrowing, ColBERTv2 reranking, and a global hybrid score to improve result order.

rag-retrieval (knowledge_retrieval)

The search engine is designed to serve as retrieval infrastructure for RAG systems and Pair Chat Assistants.

query-rewriting (knowledge_retrieval)

The team is exploring LLM-based query expansion to append related terms and improve retrieval/ranking.

Tradeoffs

Prioritize core search before generated summaries

Chose Core retrieval and ranking experience over Launching Magic Summary first.

Rationale: The team held back summarization to focus on search quality and usefulness first.

Use e5 embeddings

Chose e5 embeddings over OpenAI ada embeddings.

Rationale: e5 offered a balance of speed, cost-effectiveness, and performance for this search workload.

Use hybrid scoring

Chose Semantic, keyword, and ColBERTv2 hybrid score over A single ranking metric.

Rationale: Hybrid scoring reduced bias toward any one dimension of result quality.

Gotchas (what bit them)

Keyword frequency can swamp phrase intent.

Root cause: A purely keyword-based engine ranks documents that mention one frequent term many times above documents that answer the full query.

Resolution: Use hybrid semantic and keyword retrieval plus reranking.

Historical government data has inconsistent formats.

Root cause: Hansard records span decades and evolving publication formats.

Resolution: Scrape, parse, and standardize reports into a uniform search format before indexing.

Complex reranking can threaten search latency.

Root cause: State-of-the-art rerankers are more resource-intensive than first-stage retrieval.

Resolution: Use a three-phase approach that narrows cheaply before applying ColBERTv2 and global scoring.

Evaluation

Pair Search evaluates quality through user feedback and search-performance metrics such as clicked-result rank and the number of pages users need before finding useful results.

Metrics: average rank of clicked results · number of pages before users find desired results · daily users · daily searches · qualitative policy-officer feedback

Outcomes

• Pair Search indexed the full Hansard database from 1955 to January 2024.
• The soft launch reached roughly 150 daily users and 200 daily searches.
• Initial policy-officer feedback reported faster and more useful search.
• Multiple agencies expressed interest in applying the core search engine to their own data.

The source reports over 30,000 Hansard reports from 1955 to January 2024, around 150 daily users, around 200 daily searches, and use by policy/legal/public-sector users in a soft launch.