← Resources & Methods
Natural Language Processing › Resources & Methods ›

Pretraining

72 directly classified papers

Papers per year

Papers

ERNIE-Doc: A Retrospective Long-Document Modeling Transformer ACL 2021
Superbizarre Is Not Superb: Derivational Morphology Improves BERT’s Interpretation of Complex Words ACL 2021
nmT5 - Is parallel data still relevant for pre-training massively multilingual language models? ACL 2021
On the Influence of Masking Policies in Intermediate Pre-training EMNLP 2021
Data Efficient Masked Language Modeling for Vision and Language EMNLP 2021
Not All Models Localize Linguistic Knowledge in the Same Place: A Layer-wise Probing on BERToids’ Representations EMNLP 2021
Revisiting Pre-Trained Models for Chinese Natural Language Processing EMNLP 2020
Getting the ##life out of living: How Adequate Are Word-Pieces for Modelling Complex Morphology? ACL 2020
TaPas: Weakly Supervised Table Parsing via Pre-training ACL 2020
Which *BERT? A Survey Organizing Contextualized Encoders EMNLP 2020
Analyzing Individual Neurons in Pre-trained Language Models EMNLP 2020
MPNet: Masked and Permuted Pre-training for Language Understanding NIPS 2020
Do Language Embeddings capture Scales? EMNLP 2020
Cost-effective Selection of Pretraining Data: A Case Study of Pretraining BERT on Social Media EMNLP 2020
SciWING– A Software Toolkit for Scientific Document Processing EMNLP 2020
Explicit Cross-lingual Pre-training for Unsupervised Machine Translation EMNLP 2019
UER: An Open-Source Toolkit for Pre-training Models EMNLP 2019
SciBERT: A Pretrained Language Model for Scientific Text EMNLP 2019
Knowledge Enhanced Contextual Word Representations EMNLP 2019
BERT Rediscovers the Classical NLP Pipeline ACL 2019
Multi-News: A Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Model ACL 2019
Pretraining Sentiment Classifiers with Unlabeled Dialog Data ACL 2018