2021
EMNLP
EMNLP 2021
What is SemEval evaluating? A Systematic Analysis of Evaluation Campaigns in NLP
Abstract
AbstractSemEval is the primary venue in the NLP community for the proposal of new challenges and for the systematic empirical evaluation of NLP systems. This paper provides a systematic quantitative analysis of SemEval aiming to evidence the patterns of the contributions behind SemEval. By understanding the distribution of task types, metrics, architectures, participation and citations over time we aim to answer the question on what is being evaluated by SemEval.
❓
The Questioner
🌉
Interdisciplinary Bridge
— Machine Learning and Natural Language Processing
🧭
Keyword Pioneer
— systematic analysis
🐣
Hot Topic Early Bird
— evaluation benchmark
🐝
Cross-Pollinator
— Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Security & Privacy, Speech & Audio
Authors
Topics
Machine Learning > Optimization & Theory > Theory
Natural Language Processing > Applications > Information Retrieval
Natural Language Processing > Applications > Text Classification
Natural Language Processing > Applications > Natural Language Inference
Machine Learning > Optimization & Theory > Evaluation