Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates

Preliminary code release for our paper "Revisiting the Adam-SGD Gap in LLM Pre-Training: The Role of Large Effective Learning Rates", by Athanasios Glentis, Dawei Li, Chung-Yiu Yau and Mingyi Hong.

Dataset

Follow c4.sh to download the C4 dataset.

Ray

To activate ray run:

ray start --head

Scripts

We provide the scripts to reproduce our LLaMA results in the scripts folder:

• train_{model_size}_SGD_LL.sh for 130M, 350M and 1B model sizes.