Math and Architectures of Deep Learning

Brief contents

• An overview of machine learning and deep learning
• Vectors, matrices, and tensors in machine learning
• Classifiers and vector calculus
• Linear algebraic tools in machine learning
• Probability distributions in machine learning
• Bayesian tools for machine learning
• Function approximation: How neural networks model the world
• Training neural networks: Forward propagation and backpropagation
• Loss, optimization, and regularization
• Convolutions in neural networks
• Neural networks for image classification and object detection
• Manifolds, homeomorphism, and neural networks
• Fully Bayes model parameter estimation
• Latent space and generative modeling, autoencoders, and variational autoencoders

Contents

• Foreword
• Preface
• Acknowledgments
• About this book

Chapter 11: Neural networks for image classification and object detection

11.1 CNNs for image classification: LeNet

PyTorch: Implementing LeNet for image classification on MNIST

11.2 Toward deeper neural networks

VGG (Visual Geometry Group) Net

Inception: Network-in-network paradigm

ResNet: Why stacking layers to add depth does not scale

PyTorch Lightning

11.3 Object detection: A brief history

R-CNN

Fast R-CNN

Faster R-CNN

11.4 Faster R-CNN: A deep dive

Convolutional backbone

Region proposal network

Fast R-CNN

Training the Faster R-CNN

Other object-detection paradigms

Chapter 12: Manifolds, homeomorphism, and neural networks

12.1 Manifolds

Hausdorff property

Second countable property

12.2 Homeomorphism

12.3 Neural networks and homeomorphism between manifolds

Chapter 13: Fully Bayes model parameter estimation

13.1 Fully Bayes estimation: An informal introduction

Parameter estimation and belief injection

13.2 MLE for Gaussian parameter values (recap)

13.3 Fully Bayes parameter estimation: Gaussian, unknown mean, known precision

13.4 Small and large volumes of training data, and strong and weak priors

13.5 Conjugate priors

13.6 Fully Bayes parameter estimation: Gaussian, unknown precision, known mean

Estimating the precision parameter

13.7 Fully Bayes parameter estimation: Gaussian, unknown mean, unknown precision

Normal-gamma distribution

Estimating the mean and precision parameters

13.8 Example: Fully Bayesian inferencing

Maximum likelihood estimation

Bayesian inference

13.9 Fully Bayes parameter estimation: Multivariate Gaussian, unknown mean, known precision

13.10 Fully Bayes parameter estimation: Multivariate, unknown precision, known mean

Wishart distribution

Estimating precision

Chapter 14: Latent space and generative modeling, autoencoders, and variational autoencoders

14.1 Geometric view of latent spaces

14.2 Generative classifiers

14.3 Benefits and applications of latent-space modeling

14.4 Linear latent space manifolds and PCA

PyTorch code for dimensionality reduction using PCA

14.5 Autoencoders

Autoencoders and PCA

14.6 Smoothness, continuity, and regularization of latent spaces

14.7 Variational autoencoders

Geometric overview of VAEs

VAE training, losses, and inferencing

VAEs and Bayes’ theorem

Stochastic mapping leads to latent-space smoothness

Direct minimization of the posterior requires prohibitively expensive normalization

ELBO and VAEs

Choice of prior: Zero-mean, unit-covariance Gaussian

Reparameterization trick

Appendix

Notations

Index

Foreword

As a lifelong student of the business of technological innovation, I have often wondered: what sets apart an expert from regular practitioners in any area of technology? An expert tends to have many micro-insights into the subject that often elude the ordinary practitioner. This enables them to come up with solutions that are not visible to others. The primary appeal of this book is to generate that kind of micro-intuitions into the complex subject of machine learning. For all their ubiquitousness, episodic internet recipes do not build such intuitions in a systematic, connected way. This book does.

I also agree with the author’s position that such intuitions are impossible to build without a firm grasp of the mathematical understanding of the core principles of machine learning. Of course, all this has to be combined with programming knowledge, without which it becomes idle theory. I like the way this book attends to both theory and practice of machine learning by presenting the mathematics alongside PyTorch code snippets.

At present, deep learning is indeed shaping human history. Machine learning and data science jobs are consistently rated as the best. If you are looking for a rewarding career in technology, this may be the area for you. And if you are looking for a book that gives you expert-level understanding but only assumes fairly basic knowledge of mathematics and programming, this is your book. With its joint, side-by-side treatment of math and PyTorch programming, it is perfect for professionals who want to become serious practitioners of the art and science of machine learning. Machine learning lies at the confluence of linear algebra, multivariate statistics, and Python programming, and this book combines them into a single coherent narrative—starting from the basics but rapidly moving into advanced topics.

A particularly delightful aspect of the book is how it creates geometric intuitions behind complex mathematical concepts. Symbols may be forgotten, but the picture remains in the head.

—PRITH BANERJEE, Chief Technology Officer ANSYS, Inc., ex Senior Vice President of Research and Director, HP Labs, formerly Professor and Director of Computational Science and Engineering, University of Illinois at Urbana-Champaign

Preface

Artificial intelligence (machine learning or deep learning to insiders) is quite the rage at this point of time. Media is full of eager and/or paranoid predictions about a world governed by this new technology and quite justifiably so. It’s a knowledge revolution happening in front of our very eyes.

Working on computer vision and image processing problems for decades for my PhD, then at Adobe Systems, then at Google, and then at Drishti Technologies (the Silicon Valley start-up that I co-founded), I have been at the bleeding edge of this revolution for a long time. I’ve seen not only what works, but also—perhaps more importantly—what does not work and what almost works. This gives me a unique perspective. Often when trying to solve practical problems, none of the textbook theories will work directly. We must mix various ideas to create a winning concoction. This requires a feel for what works and why and what doesn’t work and why. It is this feel, this understanding of the inner workings of the machine/deep learning theory, along with the insights and intuitions that I hope to transmit to my readers.

This brings me to another point. Because of the popularity of the subject, a large volume of “deep-learning-made-easy”-type material exists in print and/or online. These articles don’t do justice to the subject. My reaction to them is “everything should be made as simple as possible, but not simpler.” Deep learning can’t be learned by going through a small fragmented set of simplified recipes from which all math has been scrubbed out. This is a mathematical topic and mastery requires understanding the math along with the programming. What is needed is a resource which presents this

PREFACE

topic with the requisite amount of math—no more and no less—with the connection between the deep learning and math explicitly spelled out. This is exactly what this book strives to provide with its dual presentation of the math and corresponding PyTorch code snippets.

Acknowledgments

To all the reviewers: Al Krinker, Atul Saurav, Bobby Filar, Chris Giblin, Ekkehard Schnoor, Erik Hansson, Gaurav Bhardavj, Grigory Sapunov, Ian Graves, James J. Byleckie, Jeff Neumann, Jehad Nasser, Juan Jose Rubio Guillamon, Julien Pohie, Kevin Cheung, Krzysztof Kamyczek, Lucian Mircea Sasu, Matthias Busch, Mike Wall, Mortaza Doulaty, Morteza Kiadi, Nelson González, Nicole Königstein, Ninoslav Cˇerkez, Obiamaka Agbaneje, Pejvak Moghimi, Peter Morgan, Rauhsan Jha, Sean T. Booker, Sebastián Palma Mardones, Stefano Ongarello, Tony Holdroyd, Vishwesh Ravi Shrimali, and Wiebe de Jong, your suggestions helped make this a better book.

From Krish Chaudhury:

First and foremost, I would like to thank my family:

• Devyani (my wife), for covering my back for all these years despite an abundance of reasons not to, and for teaching me the value of pursuing excellence in whatever I do.
• Anwesa (my daughter), who fills my life with indescribable joy with her love, positive attitude, and empathy.
• Gouri (my mother), for her unquestioning faith in me.
• (Late) Dr. Sujit Chaudhury (my father), for teaching me the value of insights, sincerity, and a life of letters as a goal in itself.
• I would also like to thank Dr. Vineet Gupta (my former colleague from Google) and Dr. Srayanta Mukherjee (my former colleague from Flipkart), for their valuable comments and encouragement.

From Ananya Honnedevasthana Ashok:

Writing this book has been much harder than I initially expected. It has been a massive learning experience that wouldn’t have been possible without the unwavering support of my family. In particular, I’d like to thank:

• Dr. Ashok (my father), for being a perennial role model and always being there for me.
• Jayanthi (my mother), for her unequivocal belief in me.
• Susheela (my grandmother), for her unconditional love despite chiding me for spending long hours on the book during weekends.
• I would also like to thank all my teachers, especially Dr. Viraj Kumar and Prof. N.S. Kumar, for inspiring and indoctrinating a love of learning within me.

From Sujay Narumanchi:

This book has been a labor of love, requiring more effort than I anticipated but giving me a truly fulfilling learning experience that I will forever cherish. My family and friends have been my pillars of strength throughout this journey. I’d like to thank:

• Sivakumar (my father), for always believing in me and encouraging me to pursue my dreams.
• Vinitha (my mother), for being my rock and providing unwavering support throughout my life.
• Prabhu (my brother), for being a constant source of fun and wisdom.
• (Late) Ramachandran (my grandfather), for instilling in me a love of mathematics and teaching me the value of learning from first principles.
• My friends Ambika, Anoop, Bharat, Neel, Pranav, and Sanjana, for providing a listening ear and a shoulder to lean on.

From Devashish Shankar:

I would like to begin by thanking my parents, Dr. Shiv Shanker and Dr. Sadhana Shanker, for their unwavering support, love, and guidance. Additionally, I would like to honor the memory of my late grandfather, Dr. Ajai Shanker, who instilled in me a deep sense of curiosity and a passion for scientific thinking that has guided me throughout my life. I am also deeply grateful to my mentors and colleagues for their guidance and support.

About this book

Are you the type of person who wants to know why and how things work? Instead of feeling satisfied, even grateful, that a tool solves the problem at hand, do you try to understand what the tool is really doing, why it behaves a certain way, and whether it will work under different circumstances? If yes, you have our sympathy—life won’t be peaceful for you. You also have our best wishes—these pages are dedicated to you.

The internet abounds with prebuilt deep learning models and training systems that hardly require you to understand the underlying principles. But practical problems often do not fit any of the publicly available models. These situations call for the development of a custom model architecture. Developing such an architecture requires understanding the mathematical underpinnings of optimization and machine learning.

Deep learning and computer vision are very practical subjects, so these questions are relevant: “Is the math necessary? Shouldn’t we spend the time learning, say, the Python nuances of deep learning?” Well, yes and no. Programming skills (in particular, Python) are mandatory. But without an intuitive understanding of the mathematics, the how and why and the answer to “Can I repurpose this model?” will not be visible to you. Mathematics allows you to see the abstractions behind the implementation.

In many ways, the ability to form abstractions is the essence of higher intelligence. Abstraction enabled early humans to divine a digging and defending tool from what was merely a sharply pointed stone to other animals. The abstraction of the description of where something is with respect to another thing fixed in the environment (aka coordinate systems and vectors) has done wonders for human civilization. Mathematics is the language for abstractions: the most precise, succinct, and unambiguous known to humankind. Hence, mathematics is absolutely necessary as a tool to study deep learning. But we must remember that it is a tool—no more and no less. The ultimate purpose of all the math in the book is to bring out the intuitions and insights that are necessary to gain expertise in the complex world of machine learning.

Another equally important tool is the programming language—we have chosen PyTorch—without which all the wisdom cannot be put to practical use. This book connects the two pillars of machine learning—mathematics and programming—via numerous code snippets typically presented together with the math. The book is accompanied by fully functional code in the GitHub repository. We expect readers to work out the math with paper and pencil and then run the code on a computer to understand the results. This book is not bedtime reading.

Having (hopefully) made a case for studying the underlying mathematical principles of deep learning and computer vision, we hasten to add that mathematical rigor is not the goal of this book. Rather, the goal is to provide mathematical (in particular, geometrical) insights that make the subject more intuitive and less like black magic. At the same time, we provide Python coding exercises and visualization aids throughout. Thus, reading this book can be regarded as learning the mathematical foundations of deep learning via geometrical examples and Python exercises.

Mastery over the material presented in this book will enable you to

• Understand state-of-the-art deep learning research papers. The book provides in-depth, intuitive explanations of some of today’s seminal papers.
• Study and understand a deep learning code base.
• Use code snippets from the book in your tasks.
• Prepare for an interview for a role as a machine learning engineer/scientist.
• Determine whether a real-life problem is amenable to machine/deep learning.
• Troubleshoot neural network quality issues.
• Identify the right neural network architecture to solve a real-life problem.
• Quickly implement a prototype architecture and train a deep learning model for a real-life problem.

A word of caution: we often start with the basics but quickly go deeper. It’s important to read individual chapters from beginning to end, even if you’re familiar with the material presented at the start.

Finally, the ultimate justification for an intellectual endeavor is to have fun pursuing it. So, the authors will consider themselves successful if you enjoy reading this book.

Who should read this book?

This book is aimed toward the reader with a basic understanding of engineering mathematics and Python programming, with a serious intent to learn deep learning. For maximum benefit, the math should be worked out with paper and pencil and the PyTorch programs executed on a computer. Here are some possible reader profiles:

• A person with a degree in engineering, science, or math, possibly acquired a while ago, who is considering a career switch to deep learning. No prior knowledge of machine learning or deep learning is required.
• An entry- or mid-level machine learning practitioner who wants to gain deeper insights into the workings of various techniques and graduate from downloading models from the internet and trying them out to developing custom deep learning solutions for real problems, and/or develop the ability to read and understand research publications on the topic.
• A college student embarking on a career of deep learning.

How this book is organized: A road map

This book consists of 14 chapters and an appendix. In general, all mathematical concepts are examined from a machine learning point of view. Geometric insights are brought out and PyTorch code is provided wherever appropriate.

• Chapter 1 is an overview of machine learning and deep learning. Its purpose is to establish the big picture context in the reader’s mind and familiarize the reader with some machine learning concepts like input space, feature space, model training, architecture, loss, and so on.
• Chapter 2 covers the core concepts of vectors and matrices which form the building blocks for machine learning. It introduces the notions of dot product, vector length, orthogonality, linear systems, eigenvalues and eigenvectors, Moore-Penrose pseudo inverse, matrix diagonalization, spectral decomposition, and so on.
• Chapter 3 provides an overview of vector calculus concepts needed for understanding deep learning. We introduce gradients, local approximation of multidimensional functions via Taylor expansion in arbitrary dimensional spaces, Hessian matrices, gradient descent, convexity, and the connection of all these with the idea of loss minimization in machine learning. This chapter provides the first taste of PyTorch model building.
• Chapter 4 introduces principal component analysis (PCA) and singular value decomposition (SVD)—key linear algebraic tools for machine learning. We provide end-to-end PyTorch implementation of a SVD-based document retrieval system.
• Chapter 5 explains the basic concepts of probability distributions from a deep learning point of view. We look at the important properties of distributions like expected value, variance and covariance, and we also cover some of the most popular probability distributions like Gaussian, Bernoulli, binomial, multinomial, categorical, and so on. We also introduce the PyTorch distributions package.
• Chapter 6 explores Bayesian tools for machine learning. We study the Bayes theorem, understand model parameter estimation techniques like maximum likelihood estimation (MLE) and maximum a posteriori (MAP) estimation. We also look at latent variables, regularization, MLE for Gaussian distributions, entropy, cross entropy, conditional entropy, and KL divergence. We finally look at Gaussian mixture models (GMMs) and how to model and estimate the parameters of a GMM.
• Chapter 7 deep dives into neural networks. We study perceptrons, the basic building block of neural networks and how multilayered perceptrons can model arbitrary polygonal decision boundaries as well as common logic gate operations. This enables them to perform classification. We discuss Cybenko’s universal approximation theorem.
• Chapter 8 covers activation functions for neural networks, the importance and intuition behind layers. We look at forward propagation and backpropagation (with mathematical proofs) and implement a simple neural network with PyTorch. We study how to train a neural network end to end.
• Chapter 9 provides an in-depth look into various loss functions which are crucial for effective learning of neural networks. We study the math and the intuitions behind popular loss functions like cross entropy loss, regression loss, focal loss, and so on, implementing them via PyTorch. We look at geometrical insights underlying various optimization techniques like SGD, Nesterov, Adagrad, Adam, and others. Additionally, we understand why regularization is important and its relationship with MLE and MAP.
• Chapter 10 introduces convolutions, a core operator for computer vision models. We study 1D, 2D, and 3D convolution, as well as transposed convolutions and their intuitive interpretations. We also implement a simple convolutional neural network via PyTorch.
• Chapter 11 introduces various neural network architectures for image classification and object detection in images. We look at several image classification architectures in detail like LeNet, VGG, Inception, and Resnet. We also provide an in-depth study of Faster R-CNN for object detection.
• Chapter 12 explores the manifolds, the properties of manifolds like homeomorphism, Haussdorf property, and second countable property, and also how manifolds tie in with neural networks.
• Chapter 13 provides an introduction to Bayesian parameter estimation. We look at injection of prior belief into parameter estimation and how it can be used in unsupervised/semi-supervised settings. Additionally, we understand conjugate priors and the estimation of Gaussian likelihood parameters under conditions of known/unknown mean and variances.
• Chapter 14 explores latent spaces and generative modeling. We understand the geometric view of latent spaces and the benefits of latent space modeling. We take another look at PCA with this new lens, along with studying autoencoders and variational autoencoders. We study how variational autoencoders regularize the latent space and hence exhibit superior properties to autoencoders.
• The appendix covers mathematical proofs and derivations for some of the mathematical properties introduced in the chapters.

About the code

This book contains many examples of source code both in numbered listings and in line with normal text. In both cases, source code is formatted in a fixed-width font like this to separate it from ordinary text. Sometimes code is also in bold to highlight code that has changed from previous steps in the chapter, such as when a new feature adds to an existing line of code.

In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this was not enough, and listings include line-continuation markers (➥). Additionally, comments in the source code have often been removed from the listings when the code is described in the text. Code annotations accompany many of the listings, highlighting important concepts.

liveBook discussion forum