Because many parameters have wildly differing "natural" scales, it makes sense to make it possible to adjust the learning rates on a per-parameter basis.
The current way this is handled is by introducing scaling factors for key parameters where this is a problem - e.g. the probe and translation offsets - to ensure that all parameters have a "typical value" around 1.
I think we can do better! I think the best way to handle this would be to have a "learning rate factor" set per-parameter, which defaults to a sensible value. Doing this in a general way could make it easier to tune reconstructions to converge faster, as I highly suspect a major cause of slow convergence at the moment is a mismatch in typical step size.
I know that at least one member of the CXI group has done work in this direction already, I think any discussion about a sensible way to introduce this while minimizing backward compatibility issues and keeping the simple "you only need to think about very few parameters if you don't want to" philosophy in place.
Because many parameters have wildly differing "natural" scales, it makes sense to make it possible to adjust the learning rates on a per-parameter basis.
The current way this is handled is by introducing scaling factors for key parameters where this is a problem - e.g. the probe and translation offsets - to ensure that all parameters have a "typical value" around 1.
I think we can do better! I think the best way to handle this would be to have a "learning rate factor" set per-parameter, which defaults to a sensible value. Doing this in a general way could make it easier to tune reconstructions to converge faster, as I highly suspect a major cause of slow convergence at the moment is a mismatch in typical step size.
I know that at least one member of the CXI group has done work in this direction already, I think any discussion about a sensible way to introduce this while minimizing backward compatibility issues and keeping the simple "you only need to think about very few parameters if you don't want to" philosophy in place.