ICICLE

ICICLE: Expanding Retrieval with In-Context Documents Yu-Chen Den*, Yung-Yu Shih*, Zhi Rui Tam, Kuan-Yu Chen, Pu-Jen Cheng, Yun-Nung Chen, Eugene Yang (*Equal Contribution) -- National Taiwan University & Johns Hopkins University HLTCOE

Findings of the Association for Computational Linguistics: EMNLP 2026

Setup

Install uv for package management

pip install uv && uv sync

Download data

Use huggingface-hub to donwload prepared dataset

# Stage1
Abner0803/msmarco_text-with_pseudo_query-100k-gr
Abner0803/nq_text-with_pseudo_query-100k-gr

#Stage2

Training

We suggest everyone to use Axolotl to finetune our model

1. Stage-1 Finetune (Qwen3-1.7B-Base -> Generative Retrieval)

axolotl train ./configs/axolotl_qwen3-1.7b_msmarco-100k.yml
axolotl train ./configs/axolotl_qwen3-1.7b_nq-100k.yml

2. In-Context template tuning

axolotl train ./configs/axolotl_icl_msmarco-100k_v3.yml

[!NOTE] We'll need to inference this stage's checkpoint first to build the training data for DPO.

3. DPO

4. Long Context Extension

Inference

Run inference code with below command when finish training ICICLE

python -m src.inference_icl_with_tag

Citation

If you use this code, please cite the following:

@article{den2026icicle,
  title={ICICLE: Expanding Retrieval with In-Context Documents},
  author={Den, Yu-Chen and Shih, Yung-Yu and Tam, Zhi Rui and Chen, Kuan-Yu and Cheng, Pu-Jen and Chen, Yun-Nung and Yang, Eugene},
  journal={arXiv preprint arXiv:2605.26902},
  year={2026}
}