Code for this paper Training Stronger Baselines for Learning to Optimize.

Tianlong Chen*, Weiyi Zhang*, Jingyang Zhou, Shiyu Chang, Sijia Liu, Lisa Amini, Zhangyang Wang

With many efforts devoted to designing more sophisticated L2O models, we argue for another orthogonal, under-explored theme: the training techniques for those L2O models. We show that even the simplest L2O model could have been trained much better.

• Curriculum Learning

  We first present a progressive training scheme to gradually increase the optimizer unroll length, to mitigate a well-known L2O dilemma of truncation bias (shorter unrolling) versus gradient explosion (longer unrolling).

• Imitation Learning

  We further leverage off-policy imitation learning to guide the L2O learning , by taking reference to the behavior of analytical optimizers.

Our improved training techniques are plugged into a variety of state-of-the-art L2O models, and immediately boost their performance, without making any change to their model structures.

Experimental details on L2O-DM and RNNProp are refer to this README.

Experimental details on L2O-Scale are refer to this README.

If you use this code for your research, please cite our paper: