← Resources & Methods
Natural Language Processing › Resources & Methods ›

Pretraining

72 directly classified papers

Papers per year

Papers

Balaur: Language Model Pretraining with Lexical Semantic Relations EMNLP 2023
Knowledge-Selective Pretraining for Attribute Value Extraction EMNLP 2023
TRIP: Accelerating Document-level Multilingual Pre-training via Triangular Document-level Pre-training on Parallel Data Triplets EMNLP 2023
Hierarchical Pretraining on Multimodal Electronic Health Records EMNLP 2023
A Survey on Model Compression and Acceleration for Pretrained Language Models AAAI 2023
PairSpanBERT: An Enhanced Language Model for Bridging Resolution ACL 2023
SAS: Self-Augmentation Strategy for Language Model Pre-training AAAI 2022
What do Large Language Models Learn beyond Language? EMNLP 2022
SRL4E – Semantic Role Labeling for Emotions: A Unified Evaluation Framework ACL 2022
TextPruner: A Model Pruning Toolkit for Pre-Trained Language Models ACL 2022
Improving Relation Extraction through Syntax-induced Pre-training with Dependency Masking ACL 2022
About Time: Do Transformers Learn Temporal Verbal Aspect? ACL 2022
Using BERT Embeddings to Model Word Importance in Conversational Transcripts for Deaf and Hard of Hearing Users ACL 2022
AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarization EMNLP 2022
Mask More and Mask Later: Efficient Pre-training of Masked Language Models by Disentangling the [MASK] Token EMNLP 2022
CogKTR: A Knowledge-Enhanced Text Representation Toolkit for Natural Language Understanding EMNLP 2022
Pre-training Transformer Models with Sentence-Level Objectives for Answer Sentence Selection EMNLP 2022
Pre-training Language Models with Deterministic Factual Knowledge EMNLP 2022
G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks EMNLP 2022
Language Contamination Helps Explains the Cross-lingual Capabilities of English Pretrained Models EMNLP 2022
Making Pretrained Language Models Good Long-tailed Learners EMNLP 2022
BERT is to NLP what AlexNet is to CV: Can Pre-Trained Language Models Identify Analogies? ACL 2021
Convolutions and Self-Attention: Re-interpreting Relative Positions in Pre-trained Language Models ACL 2021
Syntax-Enhanced Pre-trained Model ACL 2021
ARMAN: Pre-training with Semantically Selecting and Reordering of Sentences for Persian Abstractive Summarization EMNLP 2021