2022
EMNLP
EMNLP 2022
Iterative Stratified Testing and Measurement for Automated Model Updates
Abstract
AbstractAutomating updates to machine learning systems is an important but understudied challenge in AutoML. The high model variance of many cutting-edge deep learning architectures means that retraining a model provides no guarantee of accurate inference on all sample types. To address this concern, we present Automated Data-Shape Stratified Model Updates (ADSMU), a novel framework that relies on iterative model building coupled with data-shape stratified model testing and improvement. Using ADSMU, we observed a 26% (relative) improvement in accuracy for new model use cases on a large-scale NLU system, compared to a naive (manually) retrained baseline and current cutting-edge methods.
🌉
Interdisciplinary Bridge
— Artificial Intelligence and Deep Learning and Machine Learning
🧭
Keyword Pioneer
— automated model update
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
Authors
Topics
Machine Learning > Learning Types > Continual Learning
Machine Learning > Optimization & Theory > Optimization
Machine Learning > Application Areas > Efficient Computing
Machine Learning > Application Areas > Risk Management
Machine Learning > Optimization & Theory > Evaluation
Artificial Intelligence > Core AI > Efficient Computing
Deep Learning > Optimization & Theory > Evaluation