←Back to Research
Information Retrieval

Re-ranking and Late Interaction Drive Retrieval Quality: A Controlled Comparison of RAG Strategies for Scientific Question Answering

RAGInformation RetrievalLLMsColBERTv2Agentic AI

A controlled comparison of six retrieval-augmented generation (RAG) strategies for scientific question answering, presented at the AIxSET Conference 2026.

Read Paper

Re-ranking and Late Interaction Drive Retrieval Quality

Overview

Retrieval-Augmented Generation (RAG) is now the standard way to ground Large Language Models (LLMs) in external knowledge, yet the design space of retrieval pipelines is large and the trade-offs between variants are not well understood, especially on domain-specific corpora at realistic scale.

The Study

This work presents a controlled comparison of six retrieval strategies for scientific question answering:

  1. Classic top-k dense retrieval
  2. LLM-based query rephrasing
  3. Query rephrasing followed by LLM-based reranking
  4. Multi-query fusion via Reciprocal Rank Fusion (RRF)
  5. Agentic tool-call pipeline where the generator decides for itself whether to retrieve
  6. Late-interaction retrieval with ColBERTv2

All six pipelines share the same generator (Meta-Llama/Llama-3.1-8B-Instruct), prompt, and evaluation protocol; the five single-vector pipelines additionally share SPECTER2 embeddings and a Chroma vector store; and all six retrieve from the full corpus of 463,971 arXiv papers dated 2024-2025.

Contributions

To support reproducible, large-scale evaluation, we released a synthetic question dataset of 19,484 problem-statement and methodology questions generated by Llama-3.1-8B-Instruct from a random sample of 10,000 papers across academic domains. Every strategy is evaluated on this same query set using an LLM-as-a-judge protocol along multiple quality dimensions, together with direct gold-paper retrieval metrics.

Impact

The result is an open testbed for studying the cost and quality trade-offs of RAG design choices on a research-literature corpus, and a basis for future work on faithfulness, retrieval robustness, and agentic retrieval.

Venue

Presented at the AIxSET Conference 2026.