Learn Generative AI with PyTorch

Build GANs, transformers, and diffusion models

Mark Liu

FOREWORD BY SARAH SANDERS

Contents

• Acknowledgments
• About this book
• Preface

PART 1 Introduction to generative AI

• 1 What is generative AI and why PyTorch?
  • 1.1 Introducing generative AI and PyTorch
    • What is generative AI?
    • The Python programming language
    • Using PyTorch as our AI framework
  • 1.2 GANs
    • A high-level overview of GANs
    • An illustrating example: Generating anime faces
    • Why should you care about GANs?
  • 1.3 Transformers
    • The attention mechanism
    • The Transformer architecture
    • Multimodal Transformers and pretrained LLMs
  • 1.4 Why build generative models from scratch?
• 2 PyTorch fundamentals
  • 2.1 Data types in PyTorch
    • Creating PyTorch tensors
    • Index and slice PyTorch tensors
    • PyTorch tensor shapes
    • Mathematical operations on PyTorch tensors
  • 2.2 An end-to-end deep learning project with PyTorch
    • Deep learning in PyTorch: A high-level overview
    • Preprocessing data
  • 2.3 Binary classification
    • Creating batches
    • Building and training a binary classification model
    • Testing the binary classification model
  • 2.4 Multicategory classification
    • Validation set and early stopping
    • Building and training a multicategory classification model
• 3 Generative adversarial networks: Shape and number
  • 3.1 Steps involved in training GANs
  • 3.2 Preparing training data
    • A training dataset that forms an exponential growth curve
    • Preparing the training dataset
    • Creating GANs
    • The discriminator network
    • The generator network

PART 2 Image generation

• 4 Image generation with generative adversarial networks
  • 4.1 GANs to generate grayscale images of clothing items
    • Training samples and the discriminator
    • A generator to create grayscale images
    • Training GANs to generate images of clothing items
  • 4.2 Convolutional layers
    • How do convolutional operations work?
    • How do stride and padding affect convolutional operations?
  • 4.3 Transposed convolution and batch normalization
    • How do transposed convolutional layers work?
    • Batch normalization
  • 4.4 Color images of anime faces
    • Downloading anime faces
    • Channels-first color images in PyTorch
  • 4.5 Deep convolutional GAN
    • Building a DCGAN
    • Training and using DCGAN
• 5 Selecting characteristics in generated images
  • 5.1 The eyeglasses dataset
    • Downloading the eyeglasses dataset
    • Visualizing images in the eyeglasses dataset
  • 5.2 cGAN and Wasserstein distance
    • WGAN with gradient penalty
    • cGANs
  • 5.3 Create a cGAN
    • A critic in cGAN
    • A generator in cGAN
    • Weight initialization and the gradient penalty function
  • 5.4 Training the cGAN
    • Adding labels to inputs
    • Training the cGAN
  • 5.5 Selecting characteristics in generated images
    • Selecting images with or without eyeglasses
    • Vector arithmetic in latent space
    • Selecting two characteristics simultaneously

6 CycleGAN: Converting blond hair to black hair

6.1 CycleGAN and cycle consistency loss

What is CycleGAN?

Cycle consistency loss

6.2 The celebrity faces dataset

Downloading the celebrity faces dataset

Process the black and blond hair image data

6.3 Building a CycleGAN model

Creating two discriminators

Creating two generators

6.4 Using CycleGAN to translate between black and blond hair

Training a CycleGAN to translate between black and blond hair

Round-trip conversions of black hair images and blond hair images

7 Image generation with variational autoencoders

7.1 An overview of AEs

What is an AE?

Steps in building and training an AE

7.2 Building and training an AE to generate digits

Gathering handwritten digits

Building and training an AE

Saving and using the trained AE

7.3 What are VAEs?

Differences between AEs and VAEs

The blueprint to train a VAE to generate human face images

7.4 A VAE to generate human face images

Building a VAE

Training the VAE

Generating images with the trained VAE

Encoding arithmetic with the trained VAE

PART 3 NATURAL LANGUAGE PROCESSING AND TRANSFORMERS

8 Text generation with recurrent neural networks

8.1 Introduction to RNNs

Challenges in generating text

How do RNNs work?

Steps in training a LSTM model

8.2 Fundamentals of NLP

Different tokenization methods

Word embedding

8.3 Preparing data to train the LSTM model

Downloading and cleaning up the text

Creating batches of training data

8.4 Building and training the LSTM model

Building an LSTM model

Training the LSTM model

8.5 Generating text with the trained LSTM model

Generating text by predicting the next token

Temperature and top-K sampling in text generation

9 By-line implementation of attention and Transformer

9.1 Introduction to attention and Transformer

The attention mechanism

The Transformer architecture

Different types of Transformers

9.2 Building an encoder

The attention mechanism

Creating an encoder

9.3 Building an encoder-decoder Transformer

Creating a decoder layer

Creating an encoder-decoder Transformer

9.4 Putting all the pieces together

Defining a generator

Creating a model to translate between two languages

10 Training a Transformer to translate English to French

10.1 Subword tokenization

Tokenizing English and French phrases

Sequence padding and batch creation

10.2 Word embedding and positional encoding

Word embedding

Positional encoding

10.3 Training the Transformer for English-to-French translation

Loss function and the optimizer

The training loop

10.4 Translating English to French with the trained model

11 Building a generative pretrained Transformer from scratch

11.1 GPT-2 architecture and causal self-attention

The architecture of GPT-2

Word embedding and positional encoding in GPT-2

Causal self-attention in GPT-2

11.2 Building GPT-2XL from scratch

BPE tokenization

The Gaussian error linear unit activation function

Causal self-attention

Constructing the GPT-2XL model

11.3 Loading up pretrained weights and generating text

Loading up pretrained parameters in GPT-2XL

Defining a generate() function to produce text

Text generation with GPT-2XL

19 Training a Transformer to generate text

12.1 Building and training a GPT from scratch

The architecture of a GPT to generate text

The training process of the GPT model to generate text

12.2 Tokenizing text of Hemingway novels

Tokenizing the text

Creating batches for training

12.3 Building a GPT to generate text

Model hyperparameters

Modeling the causal self-attention mechanism

Building the GPT model

12.4 Training the GPT model to generate text

Training the GPT model

A function to generate text

Text generation with different versions of the trained model

PART 4 APPLICATIONS AND NEW DEVELOPMENTS

12 Music generation with MuseGAN

13.1 Digital music representation

Musical notes, octave, and pitch

An introduction to multitrack music

Digitally represent music: Piano rolls

13.2 A blueprint for music generation

Constructing music with chords, style, melody, and groove

A blueprint to train a MuseGAN

13.3 Preparing the training data for MuseGAN

Downloading the training data

Converting multidimensional objects to music pieces

14 Building and training a music Transformer

14.1 Introduction to the music Transformer

Performance-based music representation

The music Transformer architecture

Training the music Transformer

14.2 Tokenizing music pieces

Downloading training data

Tokenizing MIDI files

Preparing the training data

14.3 Building a GPT to generate music

Hyperparameters in the music Transformer

Building a music Transformer

14.4 Training and using the music Transformer

Training the music Transformer

Music generation with the trained Transformer

15 Diffusion models and text-to-image Transformers

15.1 Introduction to denoising diffusion models

The forward diffusion process

Using the U-Net model to denoise images

A blueprint to train the denoising U-Net model

15.2 Preparing the training data

Flower images as the training data

Visualizing the forward diffusion process

15.3 Building a denoising U-Net model

The attention mechanism in the denoising U-Net model

The denoising U-Net model

Chapter 15

15.4 Training and using the denoising U-Net model

Using the trained model to generate flower images

15.5 Text-to-image Transformers

CLIP: A multimodal Transformer

Text-to-image generation with DALL-E 2

Chapter 16: Large Language Models and the LangChain Library

16.1 Content generation with the OpenAI API

Text generation tasks with OpenAI API

Code generation with OpenAI API

Image generation with OpenAI DALL-E 2

Speech generation with OpenAI API

16.2 Introduction to LangChain

The need for the LangChain library

Using the OpenAI API in LangChain

Zero-shot, one-shot, and few-shot prompting

16.3 A zero-shot know-it-all agent in LangChain

Applying for a Wolfram Alpha API Key

Creating an agent in LangChain

Adding tools by using OpenAI GPTs

Adding tools to generate code and images

16.4 Limitations and ethical concerns of LLMs

Limitations of LLMs

Ethical concerns for LLMs

Appendix A: Installing Python, Jupyter Notebook, and PyTorch

Appendix B: Minimally qualified readers and deep learning basics

Index

Foreword

I first met Mark at the PNC Innovation Summit at the University of Kentucky, at which we were both presenters. His topic was How Machines Learn. From our very first encounter, I was struck by Mark’s ability to explain complex concepts in an engaging and easy-to-understand manner. His knack for breaking down intricate ideas into digestible, relatable terms was truly impressive, and it’s a gift that he now shares through his latest book, Learn Generative AI with PyTorch.

At Native AI, where I am cofounder and chief operating officer, we are tasked with generating predictive synthetic data that is both highly accurate and robust. Mark’s exploration of techniques like temperature and top-K sampling to control the precision of AI-generated text is cutting-edge. These methods are essential for tailoring natural language processing outputs to specific use cases, a topic that will continue to grow in importance and business value.

Learn Generative AI with PyTorch is a comprehensive guide that not only introduces readers to the fascinating world of generative AI but also equips them with practical skills to build and implement their own models. Mark’s use of PyTorch as the framework of choice is a testament to its flexibility and power in developing advanced AI models. From long short-term memory models to variational autoencoders, generative adversarial networks, and Transformers, this book covers an impressive breadth of topics.

Mark’s book is an invaluable resource for anyone looking to dive into generative AI, whether they are beginners seeking to understand the basics or experienced practitioners aiming to expand their knowledge and skills. His ability to make complex topics accessible and engaging ensures that readers will come away with a solid understanding and the confidence to apply what they’ve learned.

I am honored to write the foreword for this exceptional book and am excited for the many readers who will benefit from Mark’s expertise. Learn Generative AI with PyTorch is sure to inspire and educate, paving the way for future innovations in the field of generative AI.

—Sarah Sanders, Cofounder and COO, NativeAI

Preface

My fascination with generative AI began a few years ago when I first saw models converting horse images into zebra images and Transformers producing lifelike text. This book is born out of my journey in building and understanding these models from scratch. It’s the book I wish I had during my experiments with various generative models. It begins with simple models, helping readers build foundational deep learning skills before advancing to more complex challenges. I chose PyTorch for its dynamic computational graph and clear syntax after experimenting with TensorFlow.

All generative models in this book are deep neural networks. The book starts with a comprehensive deep learning project in PyTorch, ideal for those new to the field. Each chapter is carefully structured to build upon the previous one. You’ll first learn to create basic content, such as shapes, numbers, and images using generative adversarial networks with straightforward architectures. As you progress, the complexity increases, culminating in building state-of-the-art models such as Transformers to generate text and music and diffusion models to generate high-resolution images.

On the surface, this book provides an exploration of various generative AI models. At a deeper level, the technological journey reflects how our mind works and the essence of what it means to be human. The prominence of deep neural networks in these generative models is a testament to our quest to understand and replicate the complex processes of human learning. Generative AI models, drawing inspiration from the marvels of evolutionary biology that shaped our brains, learn from the vast amount of data they encounter, much like we humans learn from the stimuli around us.

The implications of generative AI extend far beyond its practical applications. As we stand at the forefront of this technological revolution, we are compelled to re-evaluate our understanding of consciousness, life, and the very nature of human existence. The parallels between machine learning and human learning are striking. Just as generative AI operates through neural networks inspired by the human brain, our thoughts, emotions, and behaviors are the outputs of the neural networks within our body. Thus, the study of generative AI transcends technological boundaries, becoming an exploration of the human condition and the mechanisms that underlie our consciousness. The study of generative AI leads us to a profound speculation: Are humans, in essence, sophisticated generative AI models?

In that sense, generative AI is not just a tool: it is a mirror reflecting our deepest existential questions. As we continue to develop and interact with these technologies, we are not only shaping the future of artificial intelligence but also deepening our understanding of human intelligence. Ultimately, the exploration of generative AI is an exploration of ourselves, a journey into the heart of consciousness and the essence of life, challenging us to redefine what it means to be conscious, to be alive, and to be human.

Acknowledgments

Thank you to all the reviewers: Abhilash Babu, Ankit Virmani, Arpit Singh, Christopher Kottmyer, David Cronkite, Eduardo Rienzi, Erim Erturk, Francis Osei Annin, Georg Piwonka, Holger Voges, Ian Long, Japneet Singh, Karrtik Iyer, Kollin Trujillo, Michael Petrey, Mirerfan Gheibi, Nathan Crocker, Neeraj Gupta, Neha Shetty, Palak Mathur, Peter Henstock, Piergiorgio Faraglia, Rajat Kant Goel, Ramaa Vissa, Ravi Kiran Bamidi, Richard Tobias, Ruud Gijsen, Slavomir Furman, Sumit Pal, Thiago Britto Borges, Tony Holdroyd, Ursin Stauss, Vamsi Srinivas Parasa, Viju Kothuvatiparambil, and Walter Alexander Mata López, your suggestions helped make this a better book.

Finally, I want to express my deepest gratitude to my wife, Ivey Zhang, and my son, Andrew Liu, for their unwavering support throughout this journey.

About This Book

Learn Generative AI with PyTorch aims to guide you through the creation of various content (shapes, numbers, images, text, and music) from scratch. It begins with simple models, helping readers build foundational deep learning skills before advancing to more complex challenges. All generative models in this book are deep neural networks. The book starts with a comprehensive deep learning project in PyTorch, ideal for those new to the field. Each chapter is carefully structured to build upon the previous one. You’ll first create basic content like shapes, numbers, and images using generative adversarial networks with straightforward architectures. As you progress, the complexity increases, culminating in building state-of-the-art models like Transformers and diffusion models.

Who Should Read This Book?

Learn Generative AI with PyTorch is designed for machine learning enthusiasts and data scientists in various business fields who possess intermediate Python programming skills. This book aims to teach generative AI techniques for creating novel and innovative content, such as images, text, patterns, numbers, shapes, and audio, to enhance both their employers’ businesses and their own careers. While many free learning materials are available online covering individual topics, this book consolidates everything into a clear, easy-to-follow, and up-to-date format, making it an invaluable resource for anyone aspiring to become an expert in generative AI.

I assume the readers have a solid grasp of Python. You should be familiar with variable types, Python functions and classes, and the installation of third-party Python libraries and packages. If you need to brush up on these skills, the free online Python tutorial provided by W3Schools is a great resource https://www.w3schools.com/python/.

How This Book Is Organized: A Roadmap

This book has 16 chapters, organized into four parts. Part I introduces you to generative AI and deep learning with PyTorch.

• Chapter 1 explains what generative AI is and the rationale behind selecting PyTorch over other AI frameworks like TensorFlow for building generative models in this book.
• Chapter 2 uses PyTorch to create deep neural networks to perform binary and multicategory classifications so that you become well-versed in deep learning and classification tasks. The intention is to get you ready for the upcoming chapters, where you use deep neural networks in PyTorch to create various generative models.
• Chapter 3 introduces you to generative adversarial networks (GANs). You learn to use GANs to generate shapes and sequences of numbers with certain patterns.

Part II covers image generation.

• Chapter 4 discusses how to build and train GANs to generate high-resolution color images. In particular, you’ll learn to use convolutional neural networks to capture spatial features in images. You’ll also learn to use transposed convolutional layers to upsample and generate high-resolution feature maps in images.
• Chapter 5 details two ways to select characteristics in generated images. The first method involves selecting specific vectors in the latent space. The second method uses a conditional GAN, where you build and train a GAN with labeled data.
• Chapter 6 teaches you how to use a CycleGAN to translate images between two domains such as images with black hair and images with blond hair or horse images and zebra images.
• Chapter 7 explains how to generate high-resolution images using another generative model: autoencoders and their variant, variational autoencoders.

Part III dives into natural language processing and text generation.

• Chapter 8 discusses text generation with a recurrent neural network. Along the way, you learn how tokenization and word embedding work. You’ll also learn to generate text autoregressively with the trained model and how to use temperature and top-K sampling to control the creativity of the generated text.
• Chapter 9 builds a Transformer from scratch, based on the paper “Attention Is All You Need,” to translate between any two languages. You’ll implement line by line the multihead attention mechanism and an encoder-decoder Transformer.
• Chapter 10 trains the Transformer you built in chapter 9 with more than 47,000 pairs of English-to-French translations. You’ll learn to translate common English phrases to French with the trained model.
• Chapter 11 builds GPT-2XL, the largest version of GPT-2, from scratch. After that, you’ll learn how to extract the pretrained weights from Hugging Face and load them to your own GPT-2 model to generate text.
• Chapter 12 constructs a scaled-down version of the GPT model with approximately 5 million parameters so that you can train it on a regular computer. You’ll use three novels by Ernest Hemingway as the training data. The trained model can generate text in Hemingway style.

Part IV discusses some practical applications of the generative models in the book and the most recent developments in the field of generative AI.

• Chapter 13 builds and trains a MuseGAN to generate music. MuseGAN treats a piece of music as a multidimensional object akin to an image. The generator produces a complete piece of music and submits it to the critic for evaluation. The generator then modifies the music based on the critic’s feedback until it closely resembles real music from the training dataset.
• Chapter 14 takes a different approach to AI music creation. Instead of treating a piece of music as a multidimensional object, you treat it as a sequence of musical events. You’ll then apply techniques from text generation to predict the next element in a sequence.
• Chapter 15 introduces you to diffusion models, which form the foundation of all leading text-to-image Transformers (such as DALL-E 2 or Imagen). You’ll build and train a diffusion model to generate high-resolution flower images.
• Chapter 16 ends the book with a project in which you use the LangChain library to combine pretrained large language models with Wolfram Alpha and Wikipedia APIs to create a zero-shot know-it-all personal assistant.

Appendix A discusses how to install PyTorch on your computer, with or without a compute unified device architecture-enabled GPU. Appendix B provides information on what background you need in order to proceed with projects in this book and some basic concepts in deep learning such as loss functions, activation functions, and optimizers.

About the Code

This book contains many examples of source code both in numbered listings and in line with normal text. In both cases, source code is formatted in a fixed-width font like this to separate it from ordinary text. Sometimes code is also in bold to highlight code that has changed from previous steps in the chapter, such as when a new feature adds to an existing line of code.

In many cases, the original source code has been reformatted; we’ve added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this was not enough, and listings include line-continuation markers (➥). Additionally, comments in the source code have often been removed from the listings when the code is described in the text. Code annotations accompany many of the listings, highlighting important concepts.

liveBook Discussion Forum