Survey on Efficient Training of Large Neural Networks

Julia Gusak , Daria Cherniuk , Alena Shilova , Alexandr Katrutsa , Daniel Bershatsky , Xunyi Zhao , Lionel Eyraud-Dubois , Oleh Shliazhko , Denis Dimitrov , Ivan Oseledets , Olivier Beaumont

Machine Learning > Optimization & Theory > Neural Network Optimization Machine Learning > Application Areas > Efficient Computing Deep Learning > Techniques > Model Architecture

distributed training memory efficiency gradient checkpointing mixed precision training large language model