T-RAG: Tree-Based RAG Architecture for Question Answering Over Organizational Documents

qatar_computing_research_institute · Research & Academia · 2024 · source Agentic: no · promotional bias: low · source-backed · extracted with prompt 0.5.6 (schema v2)

Generated from extraction.json — do not edit by hand. Every claim cites verbatim source evidence; stated means the quote directly asserts the claim, implied means it supports it indirectly.

Problem

Domain: Question answering over private organizational governance documents

QCRI needed reliable question answering over private governance documents for a large non-profit organization under on-premise security and limited compute constraints, while standard RAG struggled with entity hierarchies embedded in organizational structure.

Constraints:

• Data security required on-premise deployment.
• Computational resources were limited.
• Entity hierarchy information did not embed well as ordinary text chunks.
• Responses needed to be reliable and not overly verbose.
• The model had to work with private organizational governance documents.

Architecture

T-RAG combines standard retrieval with a fine-tuned Llama-2 7B model and a tree representation of organizational entities. Custom entity detection identifies relevant organizational nodes, the tree generates textual context for entity-related queries, and QLoRA fine-tuning adapts the model using generated Q&A pairs.

Diagram confidence: reconstructed (reconstructed from prose).

flowchart LR
  private_governance_documents["Private governance documents"] --> vector_retrieval["Vector retrieval"]
  private_governance_documents --> q_and_a_generation["Q&A generation"]
  organizational_entity_tree["Organizational entity tree"] --> entity_detection_pipeline["Entity detection pipeline"]
  entity_detection_pipeline --> tree_context_generator["Tree context generator"]
  vector_retrieval --> fine_tuned_llama_2_7b["Fine-tuned Llama-2 7B"]
  tree_context_generator --> fine_tuned_llama_2_7b
  q_and_a_generation --> fine_tuned_llama_2_7b

Components

Data flow

A user question is processed for organizational entities. Standard retrieval supplies document chunks, while entity detection retrieves tree context that describes parent-child organizational relationships. The fine-tuned Llama-2 7B model receives both document and tree context and generates an answer.

Why this architecture

The tree layer was added because organizational hierarchy is central to many entity questions but is poorly represented by ordinary chunk retrieval or embeddings alone.

Models used: Llama-2 7B · QLoRA · spaCy

Scale

• The fine-tuning dataset contained 1,614 generated Q&A pairs.
  “Using QLoRA for parameter-efficient finetuning on 1,614 Q&A pairs generated from the source document”summary, stated
• T-RAG achieved 73% accuracy versus 56.8% for standard RAG.
  “the system achieved 73% accuracy compared to 56.8% for standard RAG and 54.1% for finetuning alone.”summary, stated
• Needle-in-a-Haystack accuracy remained 64.1% with relevant context among 10 unrelated chunks.
  “T-RAG maintaining 64.1% accuracy even with relevant context buried among 10 unrelated chunks, versus 33.3% for standard RAG.”summary, stated

Alternatives considered (roads not taken)

Standard RAG

Rejected because: It underperformed T-RAG, especially on entity-related hierarchy questions.

Fine-tuning alone

Rejected because: Fine-tuning alone achieved lower accuracy than tree-augmented retrieval.

Patterns

rag-retrieval (retrieval)

T-RAG augments document retrieval with tree-derived entity hierarchy context.

fine-tuning-adaptation (model adaptation)

The team used QLoRA to adapt Llama-2 7B on generated organizational Q&A pairs.

synthetic-data-generation (training data)

The model generated Q&A pairs from document chunks for fine-tuning data.

Tradeoffs

Add tree context to RAG.

Chose Tree-RAG over Standard vector RAG.

Rationale: The tree improved hierarchy and entity-question accuracy.

Use an on-premise fine-tuned small model.

Chose Fine-tuned Llama-2 7B with QLoRA over External hosted frontier model.

Rationale: The system had data-security and limited-compute constraints.

Gotchas (what bit them)

Entity hierarchy is hard for standard chunk retrieval.

Root cause: Organizational chart relationships are structural and may not be captured in embedded text chunks.

Resolution: Represent the hierarchy as a tree and inject textual tree context for entity questions.

Accuracy gains can come with verbosity.

Root cause: Tree-augmented context helped answer correctly but produced longer responses.

Resolution: Track conciseness alongside accuracy in evaluation.

Evaluation

The team compared T-RAG, standard RAG, and fine-tuning alone on answer accuracy, entity-specific tests, and Needle-in-a-Haystack robustness.

Metrics: Total accuracy. · Entity-related simple and complex question accuracy. · Needle-in-a-Haystack accuracy with 10 unrelated chunks. · Conciseness/verbosity.

Outcomes

• T-RAG achieved 73% total accuracy.
• Standard RAG achieved 56.8%; fine-tuning alone achieved 54.1%.
• Simple entity questions improved from 47.1% to 100% with tree context.
• Needle-in-a-Haystack accuracy remained 64.1% with relevant context buried among 10 unrelated chunks.

The case uses 1,614 generated Q&A pairs and reports evaluation over 37 total questions plus entity-specific and Needle-in-a-Haystack tests.