2020
EMNLP
EMNLP 2020
How does BERT capture semantics? A closer look at polysemous words
Abstract
AbstractThe recent paradigm shift to contextual word embeddings has seen tremendous success across a wide range of down-stream tasks. However, little is known on how the emergent relation of context and semantics manifests geometrically. We investigate polysemous words as one particularly prominent instance of semantic organization. Our rigorous quantitative analysis of linear separability and cluster organization in embedding vectors produced by BERT shows that semantics do not surface as isolated clusters but form seamless structures, tightly coupled with sentiment and syntax.
❓
The Questioner
🌉
Interdisciplinary Bridge
— Deep Learning and Machine Learning and Natural Language Processing
🧭
Keyword Pioneer
— semantic organization
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Speech & Audio
Authors
Topics
Natural Language Processing > Understanding > Semantic Analysis
Natural Language Processing > Resources & Methods > Large Language Models
Machine Learning > Learning Types > Representation Learning
Natural Language Processing > Resources & Methods > Language Modeling
Deep Learning > Models > Transformers