2022
EMNLP
EMNLP 2022
Zero-Shot Dynamic Quantization for Transformer Inference
Abstract
AbstractWe introduce a novel run-time method for significantly reducing the accuracy loss associated with quantizing BERT-like models to 8-bit integers. Existing methods for quantizing models either modify the training procedure, or they require an additional calibration step to adjust parameters that also requires a selected held-out dataset. Our method permits taking advantage of quantization without the need for these adjustments. We present results on several NLP tasks demonstrating the usefulness of this technique.
🌉
Interdisciplinary Bridge
— Artificial Intelligence and Deep Learning and Machine Learning
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
Authors
Topics
Artificial Intelligence > Core AI > Model Compression
Machine Learning > Application Areas > Efficient Computing
Machine Learning > Application Areas > Model Compression
Artificial Intelligence > Core AI > Efficient Computing
Deep Learning > Models > Transformers
Deep Learning > Optimization & Theory > Model Compression
Deep Learning > Optimization & Theory > Efficient Computing