Literature

The course material covers all exam-relevant topics in a quite self-contained manner. For more in-depth study, we recommend the following literature. Note that some of the books are rather detailed and involved, and more geared towards a larger lecture in a Master’s degree.

We recommend to buy and read at least one standard reference on ML, for BSc level this might be the James, for the MSc level the Hastie, Bishop, Murphy or Alplaydin, the Shalev-Shwartz for a mathematical entry point, and Deep Learning book written by Goodfelow, d2dl.ai for a python and DL entry.

Helpful References for Prerequisites

If you need to read up on some of the required topics (see Prerequisites), this list might help. We tried to keep it as short as possible.

• Alex J. Smola (2020): Dive into Deep Learning URL (An interactive deep learning book with code, math, and discussions Provides NumPy/MXNet, PyTorch, and TensorFlow implementations)(free HTML version)
• Goodfellow, Bengio, Courville (2016): Deep Learning URL (free HTML version)
• Awesome Deep Learning URL
• Andrej Karpathy blog URL
• Coursera Kurs “Neural Networks for Machine Learning” URL

Good Websites to have a look

• distill.pub: in-depth explanations of important concepts, worth checking out periodically for new material

Optimization / Training of NNs:

• Why Momentum Really Works: ULR
• Adam – latest trends in deep learning optimization URL
• Overview of Gradient Descent Optimization Algorithms URL
• Yes you should understand backprop URL
• A Recipe for Training Neural Networks URL

Regularization:

• Regularization for Deep Learning: A Taxonomy URL

CNNs:

• The Sobel and Laplacian Edge Detectors URL
• Keras Blog: How convolutional neural networks see the world URL
• The 9 Deep Learning Papers You Need To Know About URL
• Python based visualization repo for CNNs URL
• Computing Receptive Fields of Convolutional Neural Networks URL
• How Convolutional Neural Networks see the World URL
• Attention in Neural Networks and How to Use It URL
• Neural Networks - A Systematic Introduction (FU Berlin)URL
• Deep Learning - The Straight Dope (contains notebooks designed to teach deep learning)URL
• Computing Receptive Fields of Convolutional Neural Networks URL
• Stanford: Convolutional Neural Networks for Visual Recognition
  • Videos
  • Assignements und notes

Autoencoders/Variational Autoencoders

• PCA URL
• Introducing Variational Autoencoders (in Prose and Code) URL
• A Tutorial on Variational Autoencoders URL

Reinforcement Learning

• Statistical Reinforcement Learning (Lecture) URL
• Practical Reinforcement Learning (Course) URL

LSTMs:

• Understanding LSTM and its diagrams URL
• The most comprehensive yet simple and fun RNN/LSTM tutorial on the Internet. URL

Hyperparameter Optimization/ Neural Architecture Search

• Using Machine Learning to Explore Neural Network Architecture URL

Software

• R vs Python: Image Classification with Keras
• H20 related stuff:
  • Repository that contains the H2O presentation for Trevor Hastie and Rob Tibshirani’s Statistical Learning and Data Mining IV course in Washington, DC on October 19, 2016.
  • Deep Learning with H2O - PDF/Book

Material for Exercises

• Neural networks Exercises (Part-1) URL
• M. Deisenroth, A. Faisal, C. Ong. Mathematics for Machine Learning. URL
• L. Wassermann. All of Statistics. URL
• H. Wickham, G. Grolemund. R for Data Science. URL
• Introductory R course on datacamp.com URL

Machine Learning

• K. Kersting, C. Lampert, C. Rothkopf. Wie Maschinen Lernen. Springer, 2019. URL German, informal, intuitive introduction to ML. Lower than BSc level, maybe more targeted at pupils or a non-academic audience. Read if you want a very light-weight introduction into the field, or buy as present for relatives and friends if they ask what you are doing.
• G. James, D. Witten, T. Hastie, R. Tibshirani. An Introduction to Statistical Learning. MIT Press, 2010. URL Beginner-level introduction with applications in R. Very well suited for the BSc level.
• T. Hastie, R. Tibshirani, J. Friedman. The Elements of Statistical Learning. Springer, 2009. URL Standard reference for statistics-flavored ML.
• C. M. Bishop. Pattern Recognition and Machine Learning. Springer, 2006. URL Standard reference for ML-flavored ML.
• S. Shalev-Shwartz, S. Ben-David. Understanding machine learning: From Theory to Algorithms. Cambridge University Press, 2014. URL Great, thorough introduction to ML theory. Math-style book with definitions and proofs.
• E. Alpaydin. Introduction to Machine Learning. MIT Press, 2010. URL Standard reference with broad coverage; easy to read.
• K. Murphy. Machine Learning: a Probabilistic Perspective. MIT Press, 2012. URL Standard reference; quite extensive; statistical/probabilistic lens.
• F. Provost, T. Fawcett. Data Science for Business. O’Reilly, 2013. URL A very good, applied and easy-to-read book by 2 well-known ML scientists. Contains many practical aspects that are missing in other references. Probably a good idea to read this in any case.
• N. Japkowicz. Evaluating Learning Algorithms (A Classification Perspective). Cambridge University Press, 2011. Nice reading on performance measures, resampling methods and (some) statistical tests for benchmarking in ML; only for classification.
• B. Bischl et al. Hyperparameter Optimization: Foundations, Algorithms, Best Practices and Open Challenges. arXiv preprint 2021. URL Our tutorial paper on HPO.
• I. Goodfellow, Y. Bengio, A. Courville. Deep Learning. MIT Press, 2016. URL Standard, modern reference for DL.

Mathematical & Statistical Theory

• G. Strang. Linear Algebra and Learning from Data. Cambridge University Press, 2019. Serious course on matrices and applied linear algebra.
• S. Axler. Linear Algebra Done Right. Springer, 2015. URL Linear Algebra from a more theoretical but still beginner-friendly perspective
• A. M. Mood, F. A. Graybill, D. C. Boes. Introduction to the Theory of Statistics, McGraw-Hill 1974. URL Beginner-friendly intro to statistics; bit on the mathy side.
• J. Watt, R. Borhani, A. Katsaggelos. Machine Learning Refined. Cambridge University Press, 2020. URL Check chapters 2-4 plus Appendix for insightful explanations and visualizations of a variety of optimization concepts.
• T. M. Cover, J. A. Thomas. Elements of Information Theory. Wiley, 2006. URL Good intro to information theory in first hundred pages, though lacking cross-connections to ML / statistics.

Python Programming

• J. VanderPlas. Python Data Science Handbook: Essential Tools for working with Data. 2016. Or use the online website such as:
• Python Programming URL
• Python Tutorial URL

R Programming

• N. Matloff. The Art of R Programming. No Starch Press, 2011. URL

We use the mlr3 package for machine learning in R quite heavily.

• Central project page and learning resources: https://mlr3.mlr-org.com/, in particular
  • the book,
  • the gallery, and
  • the cheatsheets.
• GitHub page: https://github.com/mlr-org/mlr3