← Back to papers
2022 NIPS NeurIPS 2022

DABS 2.0: Improved Datasets and Algorithms for Universal Self-Supervision

Abstract

Universal self-supervised (SSL) algorithms hold enormous promise for making machine learning accessible to high-impact domains such as protein biology, manufacturing, and genomics. We present DABS 2.0: a set of improved datasets and algorithms for advancing research on universal SSL. We extend the recently-introduced DABS benchmark with the addition of five real-world science and engineering domains: protein biology, bacterial genomics, multispectral satellite imagery, semiconductor wafers, and particle physics, bringing the total number of domains in the benchmark to twelve. We also propose a new universal SSL algorithm, Capri, and a generalized version of masked autoencoding, and apply both on all twelve domains---the most wide-ranging exploration of SSL yet. We find that multiple algorithms show gains across domains, outperforming previous baselines. In addition, we demonstrate the usefulness of DABS for scientific study of SSL by investigating the optimal corruption rate for each algorithm, showing that the best setting varies based on the domain. Code will be released at http://github.com/alextamkin/dabs}{http://github.com/alextamkin/dabs

🌉 Interdisciplinary Bridge — Deep Learning and Machine Learning
🧭 Keyword Pioneer — universal ssl
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Interdisciplinary, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Alex Tamkin , Gaurab Banerjee , Mohamed Owda , Vincent Liu , Shashank Rammoorthy , Noah Goodman

Topics

Machine Learning > Core Methods > Representation Learning Machine Learning > Learning Types > Self-Supervised Learning Machine Learning > Application Areas > Domain Adaptation Machine Learning > Application Areas > Domain Generalization Deep Learning > Learning Types > Self-Supervised Learning Machine Learning > Learning Paradigms > Self-Supervised Learning Deep Learning > Learning Types > Transfer Learning

Keywords

transfer learning domain adaptation self-supervised learning benchmark dataset multi-domain learning universal representation masked autoencoding universal ssl protein biology
Download PDF

Related papers

Transferring Pre-trained Multimodal Representations with Cross-modal Similarity Matching 2022
A Theoretical View on Sparsely Activated Networks 2022
Prune and distill: similar reformatting of image information along rat visual cortex and deep neural networks 2022
Matryoshka Representation Learning 2022
Off-Policy Evaluation with Deficient Support Using Side Information 2022