2017
EMNLP
EMNLP 2017
The strange geometry of skip-gram with negative sampling
Abstract
AbstractDespite their ubiquity, word embeddings trained with skip-gram negative sampling (SGNS) remain poorly understood. We find that vector positions are not simply determined by semantic similarity, but rather occupy a narrow cone, diametrically opposed to the context vectors. We show that this geometric concentration depends on the ratio of positive to negative examples, and that it is neither theoretically nor empirically inherent in related embedding algorithms.
🌉
Interdisciplinary Bridge
— Artificial Intelligence and Deep Learning and Machine Learning and Natural Language Processing
🐣
Hot Topic Early Bird
— negative sampling
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio
Authors
Topics
Machine Learning > Core Methods > Embedding Learning
Natural Language Processing > Resources & Methods > Text Representation
Machine Learning > Learning Types > Representation Learning
Deep Learning > Learning Types > Representation Learning
Artificial Intelligence > Core AI > Natural Language Processing