Reviving #1807: Riemannian Preconditioned LoRA — coordination before drafting a PR

[smellslikeml]

Hi @BenjaminBossan — we'd like to revive the Riemannian Preconditioned LoRA integration from #1807 and check maintainer interest before opening a fresh PR.

Context

• Paper: Riemannian Preconditioned LoRA for Fine-Tuning Foundation Models adds a small r x r preconditioner to LoRA's gradient step; claim is faster / more LR-robust fine-tuning at negligible overhead.
• Reference impl: pilancilab/Riemannian_Preconditioned_LoRA (MIT license).
• Prior PR: Integrating Riemannian Preconditioner #1807 by @fangzhaozhang (May 2024). Reached your second review round on 2024-07-17 ("getting closer but still a few areas need to be improved"), then went stale and auto-closed on 2024-09-21.

Proposed scope (narrow)

• src/peft/optimizers/riemannian.py — a create_riemannian_optimizer(model, optimizer_cls, *, lr, reg, **kwargs) factory in the LoRA+ / LoRA-FA style, with the preconditioner applied via a subclass wrapper so it works with any Optimizer (AdamW, SGD, …) — addresses your issubclass comment on Integrating Riemannian Preconditioner #1807.
• src/peft/optimizers/__init__.py — one-line export.
• tests/test_riemannian_lora.py — new tests.

Round-1 review asks from #1807 folded in preemptively:

• PEP 8 class naming; docstrings replace implementation-detail strings
• No assert in code — proper TypeError / ValueError / RuntimeError
• issubclass check on optimizer_cls
• bf16-stable via float32 preconditioner computation
• DoRA compatibility (name-based lora_A / lora_B pair discovery skips the magnitude vector)

Explicitly deferring from #1807's tree:

• src/peft/tuners/lycoris_utils.py / poly/router.py — unclear necessity; will drop unless you confirm they're needed
• examples/riemannian_lora/ — happy to add here or as a follow-up
• The accidentally-committed events.out.tfevents.* file

Draft branch for review

A draft is pushed to a fork so you can see the diff before we open a PR:

Compare against main: main...smellslikeml:peft:feat/riemannian-preconditioned-lora

Verification: make style clean; pytest tests/test_riemannian_lora.py — 9/9 passing (factory, subclass permissiveness on AdamW + SGD, DoRA compatibility, error cases, exact paper formula, bf16 gradient finiteness).

Questions

1. Is there still maintainer interest in landing this?
2. Any objection to the wrapper-subclass approach (thin diff on top of optimizer_cls.step(), works with any optimizer) over Integrating Riemannian Preconditioner #1807's copy-and-modify AdamW approach?
3. Should the example workflow live in this PR or defer as a follow-up?

If greenlit, credit for the original design + review-round work goes to @fangzhaozhang in the PR body.

Disclosure: this coordination check and any subsequent PR are AI-assisted; will follow the guidelines in CLAUDE.md.

[fangzhaoz]

Thanks for interests! I'm caught up by my school work after the initial implementation, so feel free to open a fresh PR if you'd like.

[BenjaminBossan]

@smellslikeml Thanks for reviving the work, I reviewed your PR. Next time, please wait for confirmation before starting to work. This way, we can let you know, for instance, if someone is already working on it.