MuSiQue 2-Hop Baselines

Three extractive QA baselines on the MuSiQue dataset, evaluated on 2-hop questions only.

Baselines

#	Name	Context Source
1	RoBERTa-base (oracle)	All paragraphs concatenated
2	BM25 + RoBERTa	BM25-retrieved top-N paragraphs
3	Supporting Facts + RoBERTa	Gold is_supporting paragraphs only

All three baselines use the same RoBERTa-base model fine-tuned on Baseline 1 training data. Evaluation uses SQuAD-style EM and F1 (lowercase, remove articles, remove punctuation).

Setup

pip install -r requirements.txt

Data

python download_musique.py

This downloads musique_ans_v1.0_train.jsonl and musique_ans_v1.0_dev.jsonl from HuggingFace (drt/musique). Alternatively, download manually from https://github.com/StonyBrookNLP/musique and place the files in this directory.

Running

# Full run (GPU strongly recommended)
python musique_baselines.py

# CPU quick test (limits samples)
python musique_baselines.py --max_train_samples 500 --max_dev_samples 200 --num_epochs 1

# Skip training (reuse already-saved model)
python musique_baselines.py --skip_training

# Adjust BM25 retrieval size
python musique_baselines.py --bm25_top_n 3

Output

• results.json — EM and F1 scores for all three baselines
• roberta_musique_qa/ — saved fine-tuned model weights

Expected Results (approximate, 2-hop dev set)

Baseline	EM	F1
1. RoBERTa + All Paragraphs	~40–55%	~50–65%
2. BM25 (top-5) + RoBERTa	~30–45%	~40–57%
3. Gold Supporting Facts	~45–60%	~55–70%

Baseline 3 uses oracle supporting facts so it is an upper bound for retrieval-based methods. Baseline 1 uses all paragraphs (oracle context but noisier than Baseline 3). Baseline 2 tests real retrieval performance.