← Back to papers
2022 EMNLP EMNLP 2022

Fast Vocabulary Transfer for Language Model Compression

Abstract

AbstractReal-world business applications require a trade-off between language model performance and size. We propose a new method for model compression that relies on vocabulary transfer. We evaluate the method on various vertical domains and downstream tasks. Our results indicate that vocabulary transfer can be effectively used in combination with other compression techniques, yielding a significant reduction in model size and inference time while marginally compromising on performance.

🌉 Interdisciplinary Bridge — Artificial Intelligence and Deep Learning and Machine Learning
🐣 Hot Topic Early Bird — parameter efficient fine-tuning
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Leonidas Gee , Andrea Zugarini , Leonardo Rigutini , Paolo Torroni

Topics

Artificial Intelligence > Core AI > Model Compression Machine Learning > Application Areas > Knowledge Distillation Deep Learning > Optimization & Theory > Model Compression Deep Learning > Learning Types > Transfer Learning Deep Learning > Models > Language Models

Keywords

model compression parameter efficient language model parameter efficient fine-tuning inference optimization inference time vocabulary transfer
Download PDF

Related papers

Bilingual Lexicon Induction for Low-Resource Languages using Graph Matching via Optimal Transport 2022
Generative Entity Typing with Curriculum Learning 2022
Towards Reinterpreting Neural Topic Models via Composite Activations 2022
Weakly Supervised Headline Dependency Parsing 2022
Cross-modal Transfer Between Vision and Language for Protest Detection 2022