← Back to papers
2021 EACL EACL 2021

SJ_AJ@DravidianLangTech-EACL2021: Task-Adaptive Pre-Training of Multilingual BERT models for Offensive Language Identification

Abstract

AbstractIn this paper we present our submission for the EACL 2021-Shared Task on Offensive Language Identification in Dravidian languages. Our final system is an ensemble of mBERT and XLM-RoBERTa models which leverage task-adaptive pre-training of multilingual BERT models with a masked language modeling objective. Our system was ranked 1st for Kannada, 2nd for Malayalam and 3rd for Tamil.

🧭 Keyword Pioneer — task-adaptive pre-training
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Sai Muralidhar Jayanthi , Akshat Gupta

Topics

Natural Language Processing > Applications > Text Classification Natural Language Processing > Resources & Methods > Large Language Models

Keywords

ensemble learning offensive language detection masked language modeling multilingual bert task-adaptive pre-training
Download PDF

Related papers

Joint Coreference Resolution and Character Linking for Multiparty Conversation 2021
Progressively Pretrained Dense Corpus Index for Open-Domain Question Answering 2021
Crisscrossed Captions: Extended Intramodal and Intermodal Semantic Similarity Judgments for MS-COCO 2021
Representations for Question Answering from Documents with Tables and Text 2021
Gender and Racial Fairness in Depression Research using Social Media 2021