← Back to papers
2016 JMLR JMLR 2016

MLlib: Machine Learning in Apache Spark

Abstract

Apache Spark is a popular open-source platform for large-scale data processing that is well-suited for iterative machine learning tasks. In this paper we present MLlib, Spark's open- source distributed machine learning library. MLlib provides efficient functionality for a wide range of learning settings and includes several underlying statistical, optimization, and linear algebra primitives. Shipped with Spark, MLlib supports several languages and provides a high-level API that leverages Spark's rich ecosystem to simplify the development of end-to-end machine learning pipelines. MLlib has experienced a rapid growth due to its vibrant open-source community of over 140 contributors, and includes extensive documentation to support further growth and to let users quickly get up to speed. [abs] [ pdf ][ bib ] [ code ] [ webpage ] © JMLR 2016. (edit, beta)

🌉 Interdisciplinary Bridge — Computer Science and Machine Learning
🧭 Keyword Pioneer — apache spark
🐝 Cross-Pollinator — Artificial Intelligence, Computer Science, Computer Vision, Data Science & Analytics, Deep Learning, Healthcare & Medicine, Knowledge & Reasoning, Machine Learning, Mathematics & Optimization, Natural Language Processing, Reinforcement Learning, Robotics, Security & Privacy, Speech & Audio

Authors

Xiangrui Meng , Joseph Bradley , Burak Yavuz , Evan Sparks , Shivaram Venkataraman , Davies Liu , Jeremy Freeman , DB Tsai , Manish Amde , Sean Owen , Doris Xin , Reynold Xin , Michael J. Franklin , Reza Zadeh , Matei Zaharia , Ameet Talwalkar

Topics

Machine Learning > Application Areas > Efficient Computing Computer Science > Systems > Distributed Systems Machine Learning > Core Methods > Optimization

Keywords

scalable learning scalable machine learning parallel computing distributed machine learning apache spark machine learning library large-scale data processing data processing
Download PDF

Related papers

Trend Filtering on Graphs 2016
Causal Inference through a Witness Protection Program 2016
A Characterization of Linkage-Based Hierarchical Clustering 2016
How to Center Deep Boltzmann Machines 2016
Minimax Rates in Permutation Estimation for Feature Matching 2016