Deep Learning - Theory and Practice

By Sriram Ganapathy

Course Content

Basics of pattern recognition, Neural networks. Introduction to deep learning, convolutional networks, Applications in audio and image processing. All coding done in Python.

Contact Details


Instructor Sriram Ganapathy
Office Electrical Engineering C 334 (2nd Floor)
Email deeplearning doT cce2018 aT gmail doT com

Textbooks

  • “Pattern Recognition and Machine Learning”, C.M. Bishop, 2nd Edition, Springer, 2007.
  • “Neural Networks”, C.M. Bishop, 2nd Edition, Springer, 1995.
  • “Deep Learning”, I. Goodfellow, Y, Bengio, A. Courville, MIT Press, 2016. html

Pre-requisites

  • Random Process/Probablity and Statistics - Recommended Book "Probability and Statistics" by Stark and Woods. Video Lectures
  • Linear Algebra/Matrix Theory - Recommended book "Linear Algebra", by G. Strang - Author's webpage for Videos and Text
  • Basic Programming with Python - numpy based array and matrix operations.
  • Calculus

Slides


11-01-2018       
Introduction to Deep Learning Course. Examples. Roadmap of the course. slides
18-01-2018
Basics of Machine Learning - Decision and Inference Problems, Joint probability and posterior probabilities. Likelihood and priors. Loss matrix. Rule of maximum posterior probability. Loss function for regression.

slides
01-02-2018
Matrix Derivatives. Maximum Likelihood estimation and Gaussian Example. Linear Models for Classification. Perpendicular distance of a point from a surface. Logistic regression.
Ref - PRML, Bishop, Chap. 3. Appendix C

slides
08-02-2018 Logistic regression two class motivation. Posterior probability, sigmoid function, properties. Maximum likelihood for two class logistic regression. Cross entropy error for two class.
Ref - PRML, Bishop, Chap. 4, Sec. 4.2
slides
15-02-2018
Logistic regression for K classes, softmax function. Non-convex optimization (local and global minima), Gradient Descent - motivation and algorithm.
Ref - PRML, Bishop, Sec. 4.2 and NN, Bishop, Sec. 7.5

Code for Logistic Regression
22-02-2018
First Assignment posted - Due on March 04, 2018.
HW1
22-02-2018 Training and Validation data sets. Logistic Regression Code Discussion. Perceptron and 1 Hidden Layer Neural Networks. Non-linear separability with hidden layer network.
Ref - NN, Bishop. Sec. 3.5
slides
01-03-2018 Multi-layer perceptrons, type of hidden layer and output layer activations - sigmoid, tanh, relu, softmax functions. Error functions in MLPs. Backpropagation learning in MLP. MLP for logistic regression in Keras.
Ref - NN, Bishop. Sec. 4.8
slides
Code for MLP
08-03-2018
Midterm Exam in Class.
15-03-2018 Backpropagation in multi-layer deep neural networks. Universal approximation properties of single hidden layer networks. Need for depth. The trade-off between depth and width of networks. Representation learning in DNNs. Hierachical data abstractions. Example in Images
Ref - "Deep Learning", I. Goodfellow. Chap. 6
https://arxiv.org/abs/1311.2901

slides
22-03-2018 Convolutional neural networks. Kernels and convolutional operations. Maxpooling and subsampling. Backpropagation in CNN layers. CNN example for MNIST.
Ref - "Deep Learning", I. Goodfellow. Chap. 9

slides
Code for CNN
02-04-2018 Second assignment due on April 12.
HW2.pdf
05-04-2018 Recurrent neural networks, back propagation in recurrent neural networks. Different recurrent architectures - teacher forcing networks, encoder/decoder networks, bidirectional networks.
slides
12-04-2018 Vanishing gradient problem in RNNs. Long short term memory networks. Unsupervised representation learning - Restricted Boltzmann machines, Autoencoders. Discussion of mid-term exam
slides
26-04-2018
Final Exam in Class at 6pm.