2022
ACL
ACL 2022
Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model
Abstract
AbstractEnsembling is a popular method used to improve performance as a last resort. However, ensembling multiple models finetuned from a single pretrained model has been not very effective; this could be due to the lack of diversity among ensemble members. This paper proposes Multi-Ticket Ensemble, which finetunes different subnetworks of a single pretrained model and ensembles them. We empirically demonstrated that winning-ticket subnetworks produced more diverse predictions than dense networks and their ensemble outperformed the standard ensemble in some tasks when accurate lottery tickets are found on the tasks.
🌉
Interdisciplinary Bridge
— Artificial Intelligence and Deep Learning and Machine Learning
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
Authors
Topics
Artificial Intelligence > Core AI > Model Compression
Deep Learning > Techniques > Model Architecture
Deep Learning > Techniques > Pretraining
Machine Learning > Learning Types > Transfer Learning
Machine Learning > Core Methods > Ensemble Methods
Deep Learning > Optimization & Theory > Model Compression
Deep Learning > Learning Types > Ensemble Learning