Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence
Abstract
:1. Introduction
1.1. Scientific Computing and Machine Learning in Python
1.2. Optimizing Python’s Performance for Numerical Computing and Data Processing
2. Classical Machine Learning
2.1. Scikit-learn, the Industry Standard for Classical Machine Learning
2.2. Addressing Class Imbalance
2.3. Ensemble Learning: Gradient Boosting Machines and Model Combination
2.4. Scalable Distributed Machine Learning
3. Automatic Machine Learning (AutoML)
3.1. Data Preparation and Feature Engineering
3.2. Hyperparameter Optimization and Model Evaluation
3.3. Neural Architecture Search
- Entire structure: Generates the entire network from the ground-up by choosing and chaining together a set of primitives, such as convolutions, concatenations, or pooling. This is known as macro search.
- Cell-based: Searches for combinations of a fixed number of hand-crafted building blocks, called cells. This is known as micro search.
- Hierarchical: Extends the cell-based approach by introducing multiple levels and chaining together a fixed number of cells, iteratively using the primitives defined in lower layers to construct the higher layers. This combines macro and micro search.
- Morphism-based structure: Transfers knowledge from an existing well-performing network to a new architecture.
4.3. NDArray and Vectorized Operations
4.4. Interoperability
4.5. Classical Machine Learning on GPUs
4.6. Distributed Data Science and Machine Learning on GPUs
5. Deep Learning
5.1. Static Data Flow Graphs
5.2. Dynamic Graph Libraries with Eager Execution
5.3. JIT and Computational Efficiency
5.4. Deep Learning APIs
5.5. New Algorithms for Accelerating Large-Scale Deep Learning
6. Explainability, Interpretability, and Fairness of Machine Learning Models
6.1. Feature Importance
6.2. Constraining Nonlinear Models
6.3. Logic and Reasoning
6.4. Explaining with Interactive Visualizations
6.5. Privacy
6.6. Fairness
7. Adversarial Learning
8. Conclusions
Author Contributions
Funding
Acknowledgments
Conflicts of Interest
Abbreviations
AI | Artificial intelligence |
API | Application programming interface |
Autodifff | Automatic differentiation |
AutoML | Automatic machine learning |
BERT | Bidirectional Encoder Representations from Transformers model |
BO | Bayesian optimization |
CDEP | Contextual Decomposition Explanation Penalization |
Classical ML | Classical machine learning |
CNN | Convolutional neural network |
CPU | Central processing unit |
DAG | Directed acyclic graph |
DL | Deep learning |
DNN | Deep neural network |
ETL | Extract translate load |
GAN | Generative adversarial networks |
GBM | Gradient boosting machines |
GPU | Graphics processing unit |
HPO | Hyperparameter optimization |
IPC | Inter-process communication |
JIT | Just-in-time |
LSTM | long-short term memory |
MPI | Message-passing interface |
NAS | Neural architecture search |
NCCL | NVIDIA Collective Communications Library |
OPG | One-process-per-GPU |
PNAS | Progressive neural architecture search |
RL | Reinforcement learning |
RNN | Recurrent neural network |
SIMT | Single instruction multiple thread |
SIMD | Single instruction multiple data |
SGD | Stochastic gradient descent |
References
- McCulloch, W.S.; Pitts, W. A logical calculus of the ideas immanent in nervous activity. Bull. Math. Biophys. 1943, 5, 115–133. [Google Scholar] [CrossRef]
- Rosenblatt, F. The perceptron: A probabilistic model for information storage and organization in the brain. Psychol. Rev. 1958, 65, 386. [Google Scholar] [CrossRef] [PubMed] [Green Version]
- LeCun, Y.; Boser, B.; Denker, J.S.; Henderson, D.; Howard, R.E.; Hubbard, W.; Jackel, L.D. Backpropagation applied to handwritten zip code recognition. Neural Comput. 1989, 1, 541–551. [Google Scholar] [CrossRef]
- Piatetsky, G. Python Leads the 11 Top Data Science, Machine Learning Platforms: Trends and Analysis. 2019. Available online: https://www.kdnuggets.com/2019/05/poll-top-data-science-machine-learning-platforms.html (accessed on 1 February 2020).
- Biham, E.; Seberry, J. PyPy: Another version of Py. eSTREAM, ECRYPT Stream Cipher Proj. Rep. 2006, 38, 2006. [Google Scholar]
- Developers, P. How fast is PyPy? 2020. Available online: https://speed.pypy.org (accessed on 1 February 2020).
- Team, G. The State of the Octoverse 2020. Available online: https://octoverse.github.com (accessed on 25 March 2020).
- Oliphant, T.E. Python for scientific computing. Comput. Sci. Eng. 2007, 9, 10–20. [Google Scholar] [CrossRef] [Green Version]
- Virtanen, P.; Gommers, R.; Oliphant, T.E.; Haberland, M.; Reddy, T.; Cournapeau, D.; Burovski, E.; Peterson, P.; Weckesser, W.; Bright, J.; et al. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python. Nat. Methods 2020, 17, 261–272. [Google Scholar] [CrossRef] [Green Version]
- Mckinney, W. pandas: A Foundational Python Library for Data Analysis and Statistics. Python High Perform. Sci. Comput. 2011, 14, 1–9. [Google Scholar]
- Preston-Werner, T. Semantic Versioning 2.0.0. 2013. Semantic Versioning. Available online: https://semver.org/ (accessed on 26 January 2020).
- Authors, N. NumPy Receives First Ever Funding, Thanks to Moore Foundation. 2017. Available online: https://numfocus.org/blog/numpy-receives-first-ever-funding-thanks-to-moore-foundation (accessed on 1 February 2020).
- Fedotov, A.; Litvinov, V.; Melik-Adamyan, A. Speeding up Numerical Calculations in Python. 2016. Available online: http://russianscdays.org/files/pdf16/26.pdf (accessed on 1 February 2020).
- Blackford, L.S.; Petitet, A.; Pozo, R.; Remington, K.; Whaley, R.C.; Demmel, J.; Dongarra, J.; Duff, I.; Hammarling, S.; Henry, G.; et al. An updated set of basic linear algebra subprograms (BLAS). ACM Trans. Math. Softw. 2002, 28, 135–151. [Google Scholar]
- Angerson, E.; Bai, Z.; Dongarra, J.; Greenbaum, A.; McKenney, A.; Du Croz, J.; Hammarling, S.; Demmel, J.; Bischof, C.; Sorensen, D. LAPACK: A portable linear algebra library for high-performance computers. In Proceedings of the 1990 ACM/IEEE Conference on Supercomputing, New York, NY, USA, 12–16 November 1990; pp. 2–11. [Google Scholar]
- Team, O. OpenBLAS: An Optimized BLAS Library. 2020. Available online: https://www.openblas.net (accessed on 1 February 2020).
- Team, I. Python Accelerated (Using Intel® MKL). 2020. Available online: https://software.intel.com/en-us/blogs/python-optimized (accessed on 1 February 2020).
- Diefendorff, K.; Dubey, P.K.; Hochsprung, R.; Scale, H. Altivec extension to PowerPC accelerates media processing. IEEE Micro 2000, 20, 85–95. [Google Scholar] [CrossRef]
- Pedregosa, F.; Michel, V.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Vanderplas, J.; Cournapeau, D.; Pedregosa, F.; Varoquaux, G.; et al. Scikit-learn: Machine Learning in Python. J. Mach. Learn. Res. 2011, 12, 2825–2830. [Google Scholar]
- Buitinck, L.; Louppe, G.; Blondel, M.; Pedregosa, F.; Mueller, A.; Grisel, O.; Niculae, V.; Prettenhofer, P.; Gramfort, A.; Grobler, J.; et al. API design for machine learning software: Experiences from the Scikit-learn project. ar** activation for quantized neural networks. ar**+activation+for+quantized+neural+networks&author=Choi,+J.&author=Wang,+Z.&author=Venkataramani,+S.&author=Chuang,+P.I.J.&author=Srinivasan,+V.&author=Gopalakrishnan,+K.&publication_year=2018&journal=ar** the bad guys out: Protecting and vaccinating deep learning with JPEG compression. ar**+the+bad+guys+out:+Protecting+and+vaccinating+deep+learning+with+JPEG+compression&author=Das,+N.&author=Shanbhogue,+M.&author=Chen,+S.T.&author=Hohman,+F.&author=Chen,+L.&author=Kounavis,+M.E.&author=Chau,+D.H.&publication_year=2017&journal=ar**v" class='google-scholar' target='_blank' rel='noopener noreferrer'>Google Scholar]
- Raschka, S.; Kaufman, B. Machine learning and AI-based approaches for bioactive ligand discovery and GPCR-ligand recognition. ar**v 2020, ar**v:2001.06545. [Google Scholar]
- Battaglia, P.W.; Hamrick, J.B.; Bapst, V.; Sanchez-Gonzalez, A.; Zambaldi, V.; Malinowski, M.; Tacchetti, A.; Raposo, D.; Santoro, A.; Faulkner, R.; et al. Relational inductive biases, deep learning, and graph networks. ar**v 2018, ar**v:1806.01261. [Google Scholar]
- Fey, M.; Lenssen, J.E. Fast graph representation learning with PyTorch Geometric. ar**v 2019, ar**v:1903.02428. [Google Scholar]
- Law, S. STUMPY: A powerful and scalable Python library for time series data mining. J. Open Source Softw. 2019, 4, 1504. [Google Scholar] [CrossRef] [Green Version]
- Carpenter, B.; Gelman, A.; Hoffman, M.; Lee, D.; Goodrich, B.; Betancourt, M.; Brubaker, M.A.; Li, P.; Riddell, A. Stan: A probabilistic programming language. J. Stat. Softw. 2016. [Google Scholar] [CrossRef] [Green Version]
- Salvatier, J.; Wiecki, T.V.; Fonnesbeck, C. Probabilistic programming in Python using PyMC3. PeerJ Comput. Sci. 2016, 2016. [Google Scholar] [CrossRef] [Green Version]
- Tran, D.; Kucukelbir, A.; Dieng, A.B.; Rudolph, M.; Liang, D.; Blei, D.M. Edward: A library for probabilistic modeling, inference, and criticism. ar**v 2016, ar**v:1610.09787. [Google Scholar]
- Schreiber, J. Pomegranate: Fast and flexible probabilistic modeling in python. J. Mach. Learn. Res. 2017, 18, 5992–5997. [Google Scholar]
- Bingham, E.; Chen, J.P.; Jankowiak, M.; Obermeyer, F.; Pradhan, N.; Karaletsos, T.; Singh, R.; Szerlip, P.; Horsfall, P.; Goodman, N.D. Pyro: Deep universal probabilistic programming. J. Mach. Learn. Res. 2019, 20, 973–978. [Google Scholar]
- Phan, D.; Pradhan, N.; Jankowiak, M. Composable effects for flexible and accelerated probabilistic programming in NumPyro. ar**v 2019, ar**v:1912.11554. [Google Scholar]
- Broughton, M.; Verdon, G.; McCourt, T.; Martinez, A.J.; Yoo, J.H.; Isakov, S.V.; Massey, P.; Niu, M.Y.; Halavati, R.; Peters, E.; et al. TensorFlow Quantum: A Software Framework for Quantum Machine Learning. ar**v 2020, ar**v:2003.02989. [Google Scholar]
- Silver, D.; Hubert, T.; Schrittwieser, J.; Antonoglou, I.; Lai, M.; Guez, A.; Lanctot, M.; Sifre, L.; Kumaran, D.; Graepel, T.; et al. Mastering chess and Shogi by self-play with a general reinforcement learning algorithm. ar**v 2017, ar**v:1712.01815. [Google Scholar]
- Vinyals, O.; Babuschkin, I.; Czarnecki, W.M.; Mathieu, M.; Dudzik, A.; Chung, J.; Choi, D.H.; Powell, R.; Ewalds, T.; Georgiev, P.; et al. Grandmaster level in StarCraft II using multi-agent reinforcement learning. Nature 2019, 575, 350–354. [Google Scholar] [CrossRef]
- Quach, K. DeepMind Quits Playing Games with AI, Ups the Protein Stakes with Machine-Learning Code. 2018. Available online: https://www.theregister.co.uk/2018/12/06/deepmind_alphafold_games/ (accessed on 1 February 2020).
Cleverhans v3.0.1 | FoolBox v2.3.0 | ART v1.1.0 | DEEPSEC (2019) | AdvBox v0.4.1 | |
---|---|---|---|---|---|
Supported frameworks | |||||
TensorFlow | yes | yes | yes | no | yes |
MXNet | yes | yes | yes | no | yes |
PyTorch | no | yes | yes | yes | yes |
PaddlePaddle | no | no | no | no | yes |
(Evasion) attack mechanisms | |||||
Box-constrained L-BFGS [173] | yes | no | no | yes | no |
Adv. manipulation of deep repr. [180] | yes | no | no | no | no |
ZOO [181] | no | no | yes | no | no |
Virtual adversarial method [182] | yes | yes | yes | no | no |
Adversarial patch [183] | no | no | yes | no | no |
Spatial transformation attack [184] | no | yes | yes | no | no |
Decision tree attack [185] | no | no | yes | no | no |
FGSM [186] | yes | yes | yes | yes | yes |
R+FGSM [187] | no | no | no | yes | no |
R+LLC [187] | no | no | no | yes | no |
U-MI-FGSM [188] | yes | yes | no | yes | no |
T-MI-FGSM [188] | yes | yes | no | yes | no |
Basic iterative method [189] | no | yes | yes | yes | yes |
LLC/ILLC [189] | no | yes | no | yes | no |
Universal adversarial perturbation [190] | no | no | yes | yes | no |
DeepFool [191] | yes | yes | yes | yes | yes |
NewtonFool [192] | no | yes | yes | no | no |
Jacobian saliency map [193] | yes | yes | yes | yes | yes |
CW/CW2 [194] | yes | yes | yes | yes | yes |
Projected gradient descent [195] | yes | no | yes | yes | yes |
OptMargin [196] | no | no | no | yes | no |
Elastic net attack [197] | yes | yes | yes | yes | no |
Boundary attack [198] | no | yes | yes | no | no |
HopSkipJumpAttack [199] | yes | yes | yes | no | no |
MaxConf [200] | yes | no | no | no | no |
Inversion attack [201] | yes | yes | no | no | no |
SparseL1 [202] | yes | yes | no | no | no |
SPSA [203] | yes | no | no | no | no |
HCLU [204] | no | no | yes | no | no |
ADef [205] | no | yes | no | no | no |
DDNL2 [206] | no | yes | no | no | no |
Local search [207] | no | yes | no | no | no |
Pointwise attack [208] | no | yes | no | no | no |
GenAttack [209] | no | yes | no | no | no |
Defense mechanisms | |||||
Feature squeezing [210] | no | no | yes | no | yes |
Spatial smoothing [210] | no | no | yes | no | yes |
Label smoothing [210] | no | no | yes | no | yes |
Gaussian augmentation [211] | no | no | yes | no | yes |
Adversarial training [195] | no | no | yes | yes | yes |
Thermometer encoding [212] | no | no | yes | yes | yes |
NAT [213] | no | no | no | yes | no |
Ensemble adversarial training [187] | no | no | no | yes | no |
Distillation as a defense [214] | no | no | no | yes | no |
Input gradient regularization [215] | no | no | no | yes | no |
Image transformations [216] | no | no | yes | yes | no |
Randomization [217] | no | no | no | yes | no |
PixelDefend [218] | no | no | yes | yes | no |
Regr.-based classfication [219] | no | no | no | yes | no |
JPEG compression [220] | no | no | yes | no | no |
© 2020 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).
Share and Cite
Raschka, S.; Patterson, J.; Nolet, C. Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence. Information 2020, 11, 193. https://doi.org/10.3390/info11040193
Raschka S, Patterson J, Nolet C. Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence. Information. 2020; 11(4):193. https://doi.org/10.3390/info11040193
Chicago/Turabian StyleRaschka, Sebastian, Joshua Patterson, and Corey Nolet. 2020. "Machine Learning in Python: Main Developments and Technology Trends in Data Science, Machine Learning, and Artificial Intelligence" Information 11, no. 4: 193. https://doi.org/10.3390/info11040193