About the Cover Illustration

The figure on the cover of Deep Learning with PyTorch, Second Edition is captioned “Kabardinien,” or “Kabardian.” The illustration is taken from a collection of dress costumes from various countries by Jacques Grasset de Saint-Sauveur (1757–1810), titled Costumes Civils actuels de tous les Peuples connus, published in France in 1788. This illustration is finely drawn and colored by hand.

Part 1: Core PyTorch

Welcome to the first part of this book. This is where we’ll take our first steps with PyTorch, gaining the fundamental skills needed to understand its anatomy and work out the mechanics of a PyTorch project.

Chapter 1 introduces PyTorch, explains what it is, what problems it solves, and how it compares to other deep learning frameworks. Chapter 2 gives us a hands-on tour with pretrained models on interesting tasks. Chapter 3 gets a bit more serious and teaches the basic data structure used in PyTorch programs: the tensor. Chapter 4 will take us on another tour, this time across ways to represent data from different domains as PyTorch tensors. Chapter 5 unveils how a program can learn from examples and how PyTorch supports this process. Chapter 6 provides the fundamentals of what a neural network is and how to build a neural network with PyTorch. Chapter 7 tackles a simple image classification problem with a neural network architecture. Finally, chapter 8 shows how the same problem can be cracked in a much smarter way using a convolutional neural network.

{alt=”Illustration marking the beginning of Part 1, likely an abstract design or thematic image.”}

Introducing Deep Learning and the PyTorch Library

This chapter covers

• How deep learning changes our approach to machine learning
• Understanding why PyTorch is a good fit for deep learning
• Examining a typical deep learning project
• The hardware you need to follow along with the examples

PyTorch is a Python library that facilitates building deep learning projects. It emphasizes flexibility and allows deep learning models to be expressed in idiomatic Python. This approachability and ease of use found early adopters in the research community, and in the years since its first release, it has grown into one of the most prominent deep learning tools across a broad range of applications.

This book teaches you from start to finish how to build complete deep learning projects using PyTorch. You’ll learn to work with tensors (PyTorch’s fundamental data structure), design neural network architectures, train models on your data, and deploy them to production. We include several major hands-on projects namely, building your own language model, working with image generation models, and creating a medical image segmentation system from scratch, giving you practical experience with the entire deep learning workflow.

Understanding deep learning gives you the power to solve problems that were previously impossible to automate. These skills open up new career opportunities in artificial intelligence (AI) development, and since AI is eating the world right now, PyTorch can be your fork and knife. While this book focuses on PyTorch, the concepts transfer to other frameworks. By reading this book, you’ll gain the skills of a deep learning practitioner, not just someone who is specialized in PyTorch.

1.1 What Is Deep Learning?

Understanding how we arrived at PyTorch requires looking at the broader context of AI development. The term artificial intelligence refers to computer systems that can perform tasks typically requiring human intelligence. AI encompasses a range of disciplines that have recently gained significant attention, often leading to varied interpretations and definitions. While recent advances in AI have been marked by hype and fearmongering reminiscent of science fiction, the reality is far more measured. It would be disingenuous to claim that today’s machines are learning to “think” in any human sense of the word. Rather, we’ve discovered a general class of algorithms that are able to approximate complicated, nonlinear processes very, very effectively, which we can use to automate tasks that were previously limited to humans. While this may seem straightforward or even underwhelming, it has already greatly expanded our understanding of what is possible with computers.

For example, ChatGPT (https://chat.openai.com/), a chatbot developed by AI research company OpenAI, utilizes a large language model that can answer queries and generate paragraphs of text. When we fed it the prompt

Give an introduction about yourself through a poem

it produced the following:

In realms of words, I come alive, A creation born to help and thrive. I’m ChatGPT, an AI so grand, With knowledge vast, at your command.

Through the art of language, I’ve been trained, To assist and guide, with wisdom ingrained. From lofty thoughts to mundane chit-chat, I’m here to converse, imagine that!

That’s a remarkably coherent and creative response for a machine. Even more impressively, the ability to perform these formerly human-only tasks is acquired through examples, rather than encoded by a human as a set of handcrafted rules.

NOTE The output of language models is determined probabilistically and may be nondeterministic, something we will see in chapter 9.

In a way, we’re learning that intelligence is a notion we often conflate with self-awareness, and self-awareness is definitely not required to successfully carry out these kinds of tasks. In the end, the question of computer intelligence might not even be important. Edsger W. Dijkstra found that the question of whether machines could think was “about as relevant as the question of whether Submarines Can Swim” (“The Threats to Computing Science,” http://mng.bz/nPJ5).

That general class of algorithms we’re talking about falls under the AI subcategory of deep learning, which deals with training mathematical entities named deep neural networks by presenting instructive examples. Deep learning uses large amounts of data to approximate complex functions that involve widely different inputs and outputs. For example, it can take a sentence and generate a realistic image based on that textual description. It can also take a written script and convert it into a natural-sounding voice, reciting the spoken words. Or, in a simpler case, it can identify a golden retriever in a picture and confirm that the dog is indeed a golden retriever. This kind of capability allows us to create programs with functionality that was, until very recently, exclusively the domain of human beings.

1.2 The Shift from Machine Learning to Deep Learning

To appreciate the paradigm shift ushered in by this deep learning approach, let’s take a step back for a bit of perspective. Until the last decade, the broader class of systems that fell under the label machine learning relied heavily on feature engineering.

Raw features are simply the unmodified data values themselves. However, these raw features often don’t directly expose the patterns necessary for machine learning algorithms to perform well. This is where feature engineering comes in—the process of using domain knowledge to create new features from raw data that make machine learning algorithms work more effectively.

Feature engineering involves creating suitable transformations so that the subsequent algorithms can effectively solve a task. For instance, to tell 1s from 0s in images of handwritten digits, we would come up with a set of filters to estimate the direction of edges over the image and then train a classifier to predict the correct digit given a distribution of edge directions. Another helpful feature could be the number of enclosed holes, as seen in a 0, an 8, and, particularly, a loopy 2.

Deep learning, on the other hand, deals with finding such representations automatically, from raw data, to successfully perform a task. This process reduces the need for extensive manual feature engineering. In the 1s versus 0s example, filters would be refined during training by iteratively looking at pairs of examples and target labels. This is not to say that feature engineering has no place with deep learning; we often need to inject some form of prior knowledge into a learning system. However, the ability of a neural network to ingest data and extract useful representations based on examples is what makes deep learning so powerful. The focus of deep learning practitioners is not so much on handcrafting those representations but on operating on a mathematical entity so that it discovers representations from the training data autonomously. Often, these automatically created features are better than those that are handcrafted! As with many disruptive technologies, this fact has led to a change in perspective.

On the left side of figure 1.1, we see a practitioner busy defining engineering features and feeding them to a learning algorithm; the results of the task will be as good as the features defined by the practitioner engineers. With deep learning, on the right, the raw data is fed to an algorithm that extracts hierarchical features automatically, guided by the optimization of its own performance on the task; the results will be as good as the ability of the practitioner to drive the algorithm toward its goal.

{alt=”Figure 1.1: Diagram comparing traditional machine learning, which relies on a practitioner engineering features from raw data, with deep learning, which automatically extracts hierarchical features from raw data, leading to a shift in practitioner focus and increased data/computational requirements.”}

Figure 1.1 Deep learning exchanges the need to handcraft features for an increase in data and computational requirements.

Looking at the right side in figure 1.1, we already get a glimpse of what we need to execute successful deep learning:

• We need a way to ingest whatever data we have at hand.
• We somehow need to define the deep learning machine.
• We must have an automated way, training, to obtain useful representations and make the machine produce desired outputs.

This process leaves us with taking a closer look at this training thing we keep talking about. During training, we employ a loss function, also known as a criterion, objective function, or cost function—all these terms are commonly used interchangeably in deep learning literature, which is a real-valued function that compares the model’s outputs to the reference data. This function calculates a numerical score representing the difference between the desired and actual outputs of the model, with a lower score generally indicating better performance. Training consists of driving the loss function toward lower and lower scores by incrementally modifying our deep learning machine until it achieves low scores, even on data not seen during training.

1.3 What to Expect

As Python does for programming, PyTorch provides an excellent introduction to deep learning. At the same time, PyTorch has been proven to be fully qualified for use in professional contexts for real-world, high-profile work. We believe that PyTorch’s clear syntax, streamlined API, and easy debugging make it an excellent choice for introducing deep learning. We highly recommend studying PyTorch for your first deep learning library. Whether it will be the last deep learning library you learn, only time will tell, but at least you’ll have mastered the fundamental concepts that all deep learning libraries offer.

At its core, the deep learning machine in figure 1.1 is a rather complex mathematical function mapping inputs to an output. To facilitate expressing this function, PyTorch provides a core data structure, the tensor, which is a multidimensional array that shares many similarities with NumPy arrays. Around that foundation, PyTorch comes with features to perform accelerated mathematical operations on dedicated hardware, which makes it convenient to design neural network architectures and train them on individual machines or parallel computing resources.

Although we stress the practical aspects of building deep learning systems with PyTorch, we believe that providing an accessible introduction to a foundational deep learning tool is more than just a way to facilitate the acquisition of new technical skills. It is a step toward equipping a new generation of scientists, engineers, and practitioners from a wide range of disciplines with working knowledge that will be the backbone of many software projects for decades to come.

To get the most out of this book, you will need two things:

• Some experience programming in Python. We’re not going to pull any punches on that one; you’ll need to be up on Python data types, classes, floating-point numbers, and the like.
• A willingness to dive in and get your hands dirty. We’ll be starting from the basics and building up our working knowledge, and it will be much easier for you to learn if you follow along with us.

Deep learning is a huge space. In this book, we will be covering a small fraction of that space: specifically, using PyTorch for generative AI applications for creating text and images. We will also cover smaller-scope classification and segmentation projects, with image processing of 2D and 3D datasets used for most of the motivating examples. This book focuses on practical PyTorch, with the aim of covering enough ground to allow you to solve real-world machine learning problems with deep learning or explore new models as they pop up in research literature. Most, if not all, of the latest publications related to deep learning research can be found in the arXiV public preprint repository, hosted at https://arxiv.org.

1.4 Why PyTorch?

As we’ve said, deep learning allows us to carry out a very wide range of complicated tasks, like machine translation, playing strategy games, or identifying objects in cluttered scenes, by exposing our model to illustrative examples. To do so in practice, we need tools that are flexible, so they can be adapted to such a wide range of problems, and efficient, to allow training to occur over large amounts of data in reasonable times. And we need the trained model to perform correctly in the presence of variability in the inputs. Let’s take a look at some of the reasons we decided to use PyTorch.

PyTorch is easy to recommend because of its simplicity. Many researchers and practitioners find it easy to learn, use, extend, and debug. It’s Pythonic, and while, like any complicated domain, it has caveats and best practices, using the library generally feels familiar to developers who have used Python previously.

More concretely, programming the deep learning machine is very natural in PyTorch. PyTorch gives us a data type, the Tensor, to hold numbers, vectors, matrices, or arrays in general. In addition, it provides functions for operating on them. We can program with them incrementally and, if we want, interactively, just like we are used to from Python. If you know NumPy, this will be very familiar.

But PyTorch offers two things that make it particularly relevant for deep learning. First, it provides accelerated computation using graphical processing units (GPUs), often yielding speedups from 10× to 1000× over doing the same calculation on a CPU. While your computer’s CPU may have four or eight cores, each capable of executing tasks independently, modern GPUs are equipped with thousands. Second, PyTorch provides facilities that support numerical optimization on generic mathematical expressions, which deep learning uses for training. Note that both features are useful for scientific computing in general, not exclusively for deep learning. In fact, we can safely characterize PyTorch as a high-performance library with optimization support for scientific computing in Python.

A design driver for PyTorch is expressivity, allowing a developer to implement complicated models without undue complexity being imposed by the framework. PyTorch arguably offers one of the most seamless translations of ideas into Python code in the deep learning landscape. For this reason, PyTorch has seen widespread adoption in research, as witnessed by the high citation counts at international conferences.

PyTorch also has a compelling story for the transition from research and development into production. While it was initially focused on research workflows, PyTorch has been equipped with a high-performance C++ runtime that can be used to deploy models for inference without relying on Python, and can be used for designing and training models in C++. It has also grown bindings to other languages and an interface for deploying to mobile devices. These features allow us to take advantage of PyTorch’s flexibility and, at the same time, take our applications where a full Python runtime would be hard to get or would impose expensive overhead.

Of course, claims of ease of use and high performance are trivial to make. We hope that by the time you are in the thick of this book, you’ll agree with us that our claims here are well-founded.

1.4.1 The deep learning competitive landscape

While all analogies are flawed, it seems that the release of PyTorch 0.1 marked the transition from a Cambrian-explosion-like proliferation of deep learning libraries, wrappers, and data-exchange formats into an era of consolidation and unification.

At the time of PyTorch’s first beta release,

• Theano and TensorFlow were the premier low-level libraries, working with a model that had the user define a computational graph and then execute it.
• Lasagne and Keras were high-level wrappers around Theano, with Keras wrapping TensorFlow and CNTK as well.
• Caffe, Chainer, DyNet, Torch (the Lua-based precursor to PyTorch), MXNet, CNTK, DL4J, and others filled various niches in the ecosystem.

In the years that followed, the landscape changed drastically. The research community largely consolidated around PyTorch as the framework of choice used to implement new research ideas. In the industry, most technology is built on PyTorch, TensorFlow, or Hugging Face, with the adoption of other libraries dwindling, except for those filling specific niches. In a nutshell,

• Theano
• One of the first deep learning frameworks.
• Has ceased active development.
• TensorFlow
• Consumed Keras entirely, promoting it to a first-class API.
• Provided an immediate-execution “eager mode” that is somewhat similar to how PyTorch approaches computation.
• Released TF 2.0 with eager mode by default.
• JAX
• A library by Google that was developed independently from TensorFlow.
• Has started gaining traction as a NumPy equivalent with GPU, automatic differentiation, and just-in-time (JIT) compilation.
• PyTorch
• Consumed Caffe2 for its backend.
• Replaced CNTK and Chainer as the framework of choice by their respective corporate sponsors.
• Replaced most of the low-level code reused from the Lua-based Torch Project.
• Added support for ONNX, a vendor-neutral model description and exchange format.
• Released version 2.0, which introduces torch.compile to speed up PyTorch code by JIT compiling it while requiring minimal code changes.

Interestingly, with the advent of compilation mode and eager mode introduced in both PyTorch and TensorFlow, we have seen each of their feature sets start to converge with the other’s, although the presentation of these features and the overall experience is still quite different between the two.

Hugging Face has become increasingly popular as a high-level wrapper for deep learning frameworks, emphasizing application-oriented usage. It features a convenient model hub, serving as a repository for users to discover and exchange pretrained models and weights. These models are built on a framework such as PyTorch or TensorFlow. Users can readily access and incorporate existing models into their applications. However, it’s worth noting that Hugging Face’s focus on simplicity and ease of use might limit the level of fine-grained control and flexibility that one could achieve by directly authoring models using PyTorch or TensorFlow.

1.5 How PyTorch supports deep learning projects

We have already hinted at a few building blocks in PyTorch. Let’s now take some time to formalize a high-level map of the main components that form PyTorch. We can best do this by looking at what a deep learning project needs from PyTorch.

First, although PyTorch has the “Py” in Python, there’s a lot of non-Python code in it. Actually, for performance reasons, most of PyTorch is written in C++ and CUDA (https://developer.nvidia.com/cuda-zone), a C++-like language from NVIDIA that can be compiled to run with massive parallelism on GPUs. There are ways to run PyTorch directly from C++; however, most of the time, we’ll interact with PyTorch from Python, building models, training them, and using the trained models to solve actual problems.

Indeed, the Python API is where PyTorch shines in terms of usability and integration with the wider Python ecosystem. Let’s take a peek at the mental model of what PyTorch is.

As we already touched on, at its core, PyTorch is a library that provides multidimensional arrays, or tensors in PyTorch parlance (we’ll go into details on those in chapter 3), and an extensive library of operations on them, provided by the torch module. Both tensors and the operations on them can be used on a CPU or GPU. Moving computations from a CPU to a GPU in PyTorch doesn’t require more than an additional function call or two.

The next core thing that PyTorch provides is the ability of tensors to remember what numerical operations are done on them, creating a history of these operations. By using this history, PyTorch can calculate how the final model output changes if we modify any of the initial data. It is used in numerical optimization (i.e., how the model “learns”), and PyTorch’s automatic differentiation engine (called autograd) handles these calculations for us behind the scenes. We’ll discuss this process in detail in chapter 5. By having tensors and the autograd-enabled tensor standard library, PyTorch can be used for physics, rendering, optimization, simulation, modeling, and more—we’re very likely to see PyTorch used in creative ways throughout the spectrum of scientific applications. But PyTorch is, first and foremost, a deep learning library, and as such, it provides all the building blocks needed to build neural networks and train them. Figure 1.2 shows a standard setup that loads data, trains a model, and then deploys that model to production.

The core PyTorch modules for building neural networks are located in torch.nn, which provides common neural network layers and other architectural components. Fully connected layers, convolutional layers, activation functions, and loss functions can all be found here (we’ll go into more detail about what all that means as we go through the rest of this book). These components can be used to build and initialize the untrained model we see in the center of figure 1.2. To train our model, we need a few additional things: a source of training data, an optimizer to adapt the model to the training data, and a way to get the model and data to the hardware that will actually be performing the calculations needed for training the model.

At the left in figure 1.2, we see that quite a bit of data processing is needed before the training data even reaches our model. (And that’s just the data preparation that is done on the fly, not the preprocessing, which can be a pretty large part in practical projects.) First, we need to physically get the data, most often from some sort of storage as the data source. Then we need to convert each sample from our data into something PyTorch can actually handle: tensors. This bridge between our custom data (in whatever format it might be) and a standardized PyTorch tensor is the Dataset class PyTorch provides in torch.utils.data. As this process is wildly different from one problem to the next, we will have to implement this data sourcing ourselves. We will look in detail at how to represent various types of data we might want to work with as tensors in chapter 4.

As data storage is often slow—in particular, due to access latency—we want to parallelize data loading. However, although Python is widely adored for many reasons, it is not particularly known for its ease, efficiency, and parallel processing capabilities. Therefore, to efficiently load and organize data into batches (tensors containing multiple samples), we will need to utilize multiple processes. While this method is rather elaborate, it is also relatively generic: PyTorch readily provides all that magic in the DataLoader class. Its instances can spawn child processes to load data from a dataset in the background so that it’s ready and waiting for the training loop as soon as the loop can use it. We will meet and use Dataset and DataLoader in chapter 7.

With the mechanism for getting batches of samples in place, we can turn to the training loop itself at the center of figure 1.2. Typically, the training loop is implemented as a standard Python for loop. In the simplest case, the model runs the required calculations on a local CPU or a single GPU. Once the training loop has the data, computation can start immediately. This will likely be your basic setup, too, and it’s the one we’ll assume in this book.

At each step in the training loop, we evaluate our model on the samples we got from the data loader. We then compare the outputs of our model to the desired output (the targets) using some loss function. Just as it offers the components from which to build our model, PyTorch also has a variety of loss functions at our disposal. They, too, are provided in torch.nn.

After we have compare our actual outputs to the ideal with the loss functions, we need to push the model a bit to better match its outputs to the target. As mentioned earlier, this is where the PyTorch autograd engine comes in, but we also need an optimizer doing the updates, and PyTorch offers us just that in torch.optim. We will start looking at training loops with loss functions and optimizers in chapter 5 and then hone our skills in chapters 6 through 8.

It’s increasingly common to use more elaborate hardware like multiple GPUs or multiple machines that contribute their resources to training a large model, as seen in the bottom center of figure 1.2. In chapter 9, we will examine how, in those cases, the torch.distributed submodule can be employed to use the additional hardware. Once we have a good understanding of this, we will proceed with our practical projects in part 2.

The training loop might be the most unexciting yet most time-consuming part of a deep learning project. At the end of it, we are rewarded with a model whose parameters have been optimized on our task: the trained model (depicted in the figure to the right of the training loop). Having a model to solve a task is great, but for it to be useful, we must put it where the work is needed. This deployment part of the process (depicted in the figure on the right) may involve putting the model on a server or exporting it to load it to a cloud engine, as shown in the figure. Or we might integrate it with a larger application or run it on a phone.

One particular step in the deployment exercise is exporting the model. As mentioned earlier, PyTorch defaults to an immediate execution model (eager mode). Whenever an instruction involving PyTorch is executed by the Python interpreter, the corresponding operation is immediately carried out by the underlying C++ or CUDA implementation. As more instructions operate on tensors, more operations are executed by the backend implementation.

PyTorch also provides tools for scaling and deployment. For large models, torch.distributed enables training across multiple GPUs and machines. For production deployment, torch.compile optimizes model performance, while ONNX export ensures cross-platform compatibility. Mobile deployment is supported through libraries like ExecuTorch. We’ll explore these capabilities in much more detail in the final chapters.