--- title: "Learn Generative AI with PyTorch: Learn Generative Ai With Pytorch" id: "15064" type: "page" slug: "01-learn-generative-ai-with-pytorch" published_at: "2026-07-19T22:02:47+00:00" modified_at: "2026-07-20T00:41:40+00:00" url: "https://preppers-paradise.com/library/learngenerativeaiwithpytorch/01-learn-generative-ai-with-pytorch/" markdown_url: "https://preppers-paradise.com/library/learngenerativeaiwithpytorch/01-learn-generative-ai-with-pytorch.md" excerpt: "This book provides a practical guide to building generative artificial intelligence models using the PyTorch framework. Readers will learn to implement and train various advanced deep learning archite" taxonomy_category: - "AI & Machine Learning" - "Books" - "Free Teaser" taxonomy_post_tag: - "deep learning" - "gans" - "generative ai" - "image generation" - "machine learning" - "natural language processing" - "pytorch" - "text generation" - "transformers" - "variational autoencoders" --- # Learn Generative AI with PyTorch: Learn Generative Ai With Pytorch [← Learn Generative AI with PyTorch](/library/learngenerativeaiwithpytorch/) Chapter 1 of 70 · Free teaser # Learn Generative AI with PyTorch Build GANs, transformers, and diffusion models Mark Liu FOREWORD BY SARAH SANDERS ![](https://preppers-paradise.com/wp-content/uploads/x402-books/learngenerativeaiwithpytorch/_page_2_Picture_4.jpeg "Book cover: Learn Generative AI with PyTorch") ![](https://preppers-paradise.com/wp-content/uploads/x402-books/learngenerativeaiwithpytorch/_page_4_Picture_0.jpeg "Decorative image or internal branding") ## Contents * About the author * About the cover illustration * Foreword * Acknowledgments * About this book * Preface ### PART 1 Introduction to generative AI * **1 What is generative AI and why PyTorch?** * 1.1 Introducing generative AI and PyTorch * What is generative AI? * The Python programming language * Using PyTorch as our AI framework * 1.2 GANs * A high-level overview of GANs * An illustrating example: Generating anime faces * Why should you care about GANs? * 1.3 Transformers * The attention mechanism * The Transformer architecture * Multimodal Transformers and pretrained LLMs * 1.4 Why build generative models from scratch? * **2 PyTorch fundamentals** * 2.1 Data types in PyTorch * Creating PyTorch tensors * Index and slice PyTorch tensors * PyTorch tensor shapes * Mathematical operations on PyTorch tensors * 2.2 An end-to-end deep learning project with PyTorch * Deep learning in PyTorch: A high-level overview * Preprocessing data * 2.3 Binary classification * Creating batches * Building and training a binary classification model * Testing the binary classification model * 2.4 Multicategory classification * Validation set and early stopping * Building and training a multicategory classification model * **3 Generative adversarial networks: Shape and number** * 3.1 Steps involved in training GANs * 3.2 Preparing training data * A training dataset that forms an exponential growth curve * Preparing the training dataset * Creating GANs * The discriminator network * The generator network ### PART 2 Image generation * **4 Image generation with generative adversarial networks** * 4.1 GANs to generate grayscale images of clothing items * Training samples and the discriminator * A generator to create grayscale images * Training GANs to generate images of clothing items * 4.2 Convolutional layers * How do convolutional operations work? * How do stride and padding affect convolutional operations? * 4.3 Transposed convolution and batch normalization * How do transposed convolutional layers work? * Batch normalization * 4.4 Color images of anime faces * Downloading anime faces * Channels-first color images in PyTorch * 4.5 Deep convolutional GAN * Building a DCGAN * Training and using DCGAN * **5 Selecting characteristics in generated images** * 5.1 The eyeglasses dataset * Downloading the eyeglasses dataset * Visualizing images in the eyeglasses dataset * 5.2 cGAN and Wasserstein distance * WGAN with gradient penalty * cGANs * 5.3 Create a cGAN * A critic in cGAN * A generator in cGAN * Weight initialization and the gradient penalty function * 5.4 Training the cGAN * Adding labels to inputs * Training the cGAN * 5.5 Selecting characteristics in generated images * Selecting images with or without eyeglasses * Vector arithmetic in latent space * Selecting two characteristics simultaneously # 6 CycleGAN: Converting blond hair to black hair ## 6.1 CycleGAN and cycle consistency loss ### What is CycleGAN? ### Cycle consistency loss ## 6.2 The celebrity faces dataset ### Downloading the celebrity faces dataset ### Process the black and blond hair image data ## 6.3 Building a CycleGAN model ### Creating two discriminators ### Creating two generators ## 6.4 Using CycleGAN to translate between black and blond hair ### Training a CycleGAN to translate between black and blond hair ### Round-trip conversions of black hair images and blond hair images # 7 Image generation with variational autoencoders ## 7.1 An overview of AEs ### What is an AE? ### Steps in building and training an AE ## 7.2 Building and training an AE to generate digits ### Gathering handwritten digits ### Building and training an AE ### Saving and using the trained AE ## 7.3 What are VAEs? ### Differences between AEs and VAEs ### The blueprint to train a VAE to generate human face images ## 7.4 A VAE to generate human face images ### Building a VAE ### Training the VAE ### Generating images with the trained VAE ### Encoding arithmetic with the trained VAE # PART 3 NATURAL LANGUAGE PROCESSING AND TRANSFORMERS # 8 Text generation with recurrent neural networks ## 8.1 Introduction to RNNs ### Challenges in generating text ### How do RNNs work? ### Steps in training a LSTM model ## 8.2 Fundamentals of NLP ### Different tokenization methods ### Word embedding ## 8.3 Preparing data to train the LSTM model ### Downloading and cleaning up the text ### Creating batches of training data ## 8.4 Building and training the LSTM model ### Building an LSTM model ### Training the LSTM model ## 8.5 Generating text with the trained LSTM model ### Generating text by predicting the next token ### Temperature and top-K sampling in text generation # 9 By-line implementation of attention and Transformer ## 9.1 Introduction to attention and Transformer ### The attention mechanism ### The Transformer architecture ### Different types of Transformers ## 9.2 Building an encoder ### The attention mechanism ### Creating an encoder ## 9.3 Building an encoder-decoder Transformer ### Creating a decoder layer ### Creating an encoder-decoder Transformer ## 9.4 Putting all the pieces together ### Defining a generator ### Creating a model to translate between two languages # 10 Training a Transformer to translate English to French ## 10.1 Subword tokenization ### Tokenizing English and French phrases ### Sequence padding and batch creation ## 10.2 Word embedding and positional encoding ### Word embedding ### Positional encoding ## 10.3 Training the Transformer for English-to-French translation ### Loss function and the optimizer ### The training loop ## 10.4 Translating English to French with the trained model # 11 Building a generative pretrained Transformer from scratch ## 11.1 GPT-2 architecture and causal self-attention ### The architecture of GPT-2 ### Word embedding and positional encoding in GPT-2 ### Causal self-attention in GPT-2 ## 11.2 Building GPT-2XL from scratch ### BPE tokenization ### The Gaussian error linear unit activation function ### Causal self-attention ### Constructing the GPT-2XL model ## 11.3 Loading up pretrained weights and generating text ### Loading up pretrained parameters in GPT-2XL ### Defining a generate() function to produce text ### Text generation with GPT-2XL # 19 Training a Transformer to generate text ## 12.1 Building and training a GPT from scratch ### The architecture of a GPT to generate text ### The training process of the GPT model to generate text ## 12.2 Tokenizing text of Hemingway novels ### Tokenizing the text ### Creating batches for training ## 12.3 Building a GPT to generate text ### Model hyperparameters ### Modeling the causal self-attention mechanism ### Building the GPT model ## 12.4 Training the GPT model to generate text ### Training the GPT model ### A function to generate text ### Text generation with different versions of the trained model # PART 4 APPLICATIONS AND NEW DEVELOPMENTS # 12 Music generation with MuseGAN ## 13.1 Digital music representation ### Musical notes, octave, and pitch ### An introduction to multitrack music ### Digitally represent music: Piano rolls ## 13.2 A blueprint for music generation ### Constructing music with chords, style, melody, and groove ### A blueprint to train a MuseGAN ## 13.3 Preparing the training data for MuseGAN ### Downloading the training data ### Converting multidimensional objects to music pieces # 14 Building and training a music Transformer ## 14.1 Introduction to the music Transformer ### Performance-based music representation ### The music Transformer architecture ### Training the music Transformer ## 14.2 Tokenizing music pieces ### Downloading training data ### Tokenizing MIDI files ### Preparing the training data ## 14.3 Building a GPT to generate music ### Hyperparameters in the music Transformer ### Building a music Transformer ## 14.4 Training and using the music Transformer ### Training the music Transformer ### Music generation with the trained Transformer # 15 Diffusion models and text-to-image Transformers ## 15.1 Introduction to denoising diffusion models ### The forward diffusion process ### Using the U-Net model to denoise images ### A blueprint to train the denoising U-Net model ## 15.2 Preparing the training data ### Flower images as the training data ### Visualizing the forward diffusion process ## 15.3 Building a denoising U-Net model ### The attention mechanism in the denoising U-Net model ### The denoising U-Net model ## Chapter 15 ### 15.4 Training and using the denoising U-Net model #### Using the trained model to generate flower images ### 15.5 Text-to-image Transformers #### CLIP: A multimodal Transformer #### Text-to-image generation with DALL-E 2 ## Chapter 16: Large Language Models and the LangChain Library ### 16.1 Content generation with the OpenAI API #### Text generation tasks with OpenAI API #### Code generation with OpenAI API #### Image generation with OpenAI DALL-E 2 #### Speech generation with OpenAI API ### 16.2 Introduction to LangChain #### The need for the LangChain library #### Using the OpenAI API in LangChain #### Zero-shot, one-shot, and few-shot prompting ### 16.3 A zero-shot know-it-all agent in LangChain #### Applying for a Wolfram Alpha API Key #### Creating an agent in LangChain #### Adding tools by using OpenAI GPTs #### Adding tools to generate code and images ### 16.4 Limitations and ethical concerns of LLMs #### Limitations of LLMs #### Ethical concerns for LLMs ## Appendix A: Installing Python, Jupyter Notebook, and PyTorch ## Appendix B: Minimally qualified readers and deep learning basics ## Index ## Foreword I first met Mark at the PNC Innovation Summit at the University of Kentucky, at which we were both presenters. His topic was *How Machines Learn*. From our very first encounter, I was struck by Mark's ability to explain complex concepts in an engaging and easy-to-understand manner. His knack for breaking down intricate ideas into digestible, relatable terms was truly impressive, and it's a gift that he now shares through his latest book, *Learn Generative AI with PyTorch*. At Native AI, where I am cofounder and chief operating officer, we are tasked with generating predictive synthetic data that is both highly accurate and robust. Mark's exploration of techniques like temperature and top-K sampling to control the precision of AI-generated text is cutting-edge. These methods are essential for tailoring natural language processing outputs to specific use cases, a topic that will continue to grow in importance and business value. *Learn Generative AI with PyTorch* is a comprehensive guide that not only introduces readers to the fascinating world of generative AI but also equips them with practical skills to build and implement their own models. Mark's use of PyTorch as the framework of choice is a testament to its flexibility and power in developing advanced AI models. From long short-term memory models to variational autoencoders, generative adversarial networks, and Transformers, this book covers an impressive breadth of topics. Mark's book is an invaluable resource for anyone looking to dive into generative AI, whether they are beginners seeking to understand the basics or experienced practitioners aiming to expand their knowledge and skills. His ability to make complex topics accessible and engaging ensures that readers will come away with a solid understanding and the confidence to apply what they've learned. I am honored to write the foreword for this exceptional book and am excited for the many readers who will benefit from Mark's expertise. *Learn Generative AI with PyTorch* is sure to inspire and educate, paving the way for future innovations in the field of generative AI. —Sarah Sanders, Cofounder and COO, NativeAI ## Preface My fascination with generative AI began a few years ago when I first saw models converting horse images into zebra images and Transformers producing lifelike text. This book is born out of my journey in building and understanding these models from scratch. It's the book I wish I had during my experiments with various generative models. It begins with simple models, helping readers build foundational deep learning skills before advancing to more complex challenges. I chose PyTorch for its dynamic computational graph and clear syntax after experimenting with TensorFlow. All generative models in this book are deep neural networks. The book starts with a comprehensive deep learning project in PyTorch, ideal for those new to the field. Each chapter is carefully structured to build upon the previous one. You'll first learn to create basic content, such as shapes, numbers, and images using generative adversarial networks with straightforward architectures. As you progress, the complexity increases, culminating in building state-of-the-art models such as Transformers to generate text and music and diffusion models to generate high-resolution images. On the surface, this book provides an exploration of various generative AI models. At a deeper level, the technological journey reflects how our mind works and the essence of what it means to be human. The prominence of deep neural networks in these generative models is a testament to our quest to understand and replicate the complex processes of human learning. Generative AI models, drawing inspiration from the marvels of evolutionary biology that shaped our brains, learn from the vast amount of data they encounter, much like we humans learn from the stimuli around us. The implications of generative AI extend far beyond its practical applications. As we stand at the forefront of this technological revolution, we are compelled to re-evaluate our understanding of consciousness, life, and the very nature of human existence. The parallels between machine learning and human learning are striking. Just as generative AI operates through neural networks inspired by the human brain, our thoughts, emotions, and behaviors are the outputs of the neural networks within our body. Thus, the study of generative AI transcends technological boundaries, becoming an exploration of the human condition and the mechanisms that underlie our consciousness. The study of generative AI leads us to a profound speculation: Are humans, in essence, sophisticated generative AI models? In that sense, generative AI is not just a tool: it is a mirror reflecting our deepest existential questions. As we continue to develop and interact with these technologies, we are not only shaping the future of *artificial intelligence* but also deepening our understanding of *human intelligence*. Ultimately, the exploration of generative AI is an exploration of ourselves, a journey into the heart of consciousness and the essence of life, challenging us to redefine what it means to be conscious, to be alive, and to be human. ## Acknowledgments Thank you to all the reviewers: Abhilash Babu, Ankit Virmani, Arpit Singh, Christopher Kottmyer, David Cronkite, Eduardo Rienzi, Erim Erturk, Francis Osei Annin, Georg Piwonka, Holger Voges, Ian Long, Japneet Singh, Karrtik Iyer, Kollin Trujillo, Michael Petrey, Mirerfan Gheibi, Nathan Crocker, Neeraj Gupta, Neha Shetty, Palak Mathur, Peter Henstock, Piergiorgio Faraglia, Rajat Kant Goel, Ramaa Vissa, Ravi Kiran Bamidi, Richard Tobias, Ruud Gijsen, Slavomir Furman, Sumit Pal, Thiago Britto Borges, Tony Holdroyd, Ursin Stauss, Vamsi Srinivas Parasa, Viju Kothuvatiparambil, and Walter Alexander Mata López, your suggestions helped make this a better book. Finally, I want to express my deepest gratitude to my wife, Ivey Zhang, and my son, Andrew Liu, for their unwavering support throughout this journey. ## About This Book *Learn Generative AI with PyTorch* aims to guide you through the creation of various content (shapes, numbers, images, text, and music) from scratch. It begins with simple models, helping readers build foundational deep learning skills before advancing to more complex challenges. All generative models in this book are deep neural networks. The book starts with a comprehensive deep learning project in PyTorch, ideal for those new to the field. Each chapter is carefully structured to build upon the previous one. You'll first create basic content like shapes, numbers, and images using generative adversarial networks with straightforward architectures. As you progress, the complexity increases, culminating in building state-of-the-art models like Transformers and diffusion models. ### Who Should Read This Book? *Learn Generative AI with PyTorch* is designed for machine learning enthusiasts and data scientists in various business fields who possess intermediate Python programming skills. This book aims to teach generative AI techniques for creating novel and innovative content, such as images, text, patterns, numbers, shapes, and audio, to enhance both their employers' businesses and their own careers. While many free learning materials are available online covering individual topics, this book consolidates everything into a clear, easy-to-follow, and up-to-date format, making it an invaluable resource for anyone aspiring to become an expert in generative AI. I assume the readers have a solid grasp of Python. You should be familiar with variable types, Python functions and classes, and the installation of third-party Python libraries and packages. If you need to brush up on these skills, the free online Python tutorial provided by W3Schools is a great resource [https://www.w3schools.com/python/](https://www.w3schools.com/python/). ### How This Book Is Organized: A Roadmap This book has 16 chapters, organized into four parts. Part I introduces you to generative AI and deep learning with PyTorch. - Chapter 1 explains what generative AI is and the rationale behind selecting PyTorch over other AI frameworks like TensorFlow for building generative models in this book. - Chapter 2 uses PyTorch to create deep neural networks to perform binary and multicategory classifications so that you become well-versed in deep learning and classification tasks. The intention is to get you ready for the upcoming chapters, where you use deep neural networks in PyTorch to create various generative models. - Chapter 3 introduces you to generative adversarial networks (GANs). You learn to use GANs to generate shapes and sequences of numbers with certain patterns. Part II covers image generation. - Chapter 4 discusses how to build and train GANs to generate high-resolution color images. In particular, you'll learn to use convolutional neural networks to capture spatial features in images. You'll also learn to use transposed convolutional layers to upsample and generate high-resolution feature maps in images. - Chapter 5 details two ways to select characteristics in generated images. The first method involves selecting specific vectors in the latent space. The second method uses a conditional GAN, where you build and train a GAN with labeled data. - Chapter 6 teaches you how to use a CycleGAN to translate images between two domains such as images with black hair and images with blond hair or horse images and zebra images. - Chapter 7 explains how to generate high-resolution images using another generative model: autoencoders and their variant, variational autoencoders. Part III dives into natural language processing and text generation. - Chapter 8 discusses text generation with a recurrent neural network. Along the way, you learn how tokenization and word embedding work. You'll also learn to generate text autoregressively with the trained model and how to use temperature and top-K sampling to control the creativity of the generated text. - Chapter 9 builds a Transformer from scratch, based on the paper "Attention Is All You Need," to translate between any two languages. You'll implement line by line the multihead attention mechanism and an encoder-decoder Transformer. - Chapter 10 trains the Transformer you built in chapter 9 with more than 47,000 pairs of English-to-French translations. You'll learn to translate common English phrases to French with the trained model. - Chapter 11 builds GPT-2XL, the largest version of GPT-2, from scratch. After that, you'll learn how to extract the pretrained weights from Hugging Face and load them to your own GPT-2 model to generate text. - Chapter 12 constructs a scaled-down version of the GPT model with approximately 5 million parameters so that you can train it on a regular computer. You'll use three novels by Ernest Hemingway as the training data. The trained model can generate text in Hemingway style. Part IV discusses some practical applications of the generative models in the book and the most recent developments in the field of generative AI. - Chapter 13 builds and trains a MuseGAN to generate music. MuseGAN treats a piece of music as a multidimensional object akin to an image. The generator produces a complete piece of music and submits it to the critic for evaluation. The generator then modifies the music based on the critic's feedback until it closely resembles real music from the training dataset. - Chapter 14 takes a different approach to AI music creation. Instead of treating a piece of music as a multidimensional object, you treat it as a sequence of musical events. You'll then apply techniques from text generation to predict the next element in a sequence. - Chapter 15 introduces you to diffusion models, which form the foundation of all leading text-to-image Transformers (such as DALL-E 2 or Imagen). You'll build and train a diffusion model to generate high-resolution flower images. - Chapter 16 ends the book with a project in which you use the LangChain library to combine pretrained large language models with Wolfram Alpha and Wikipedia APIs to create a zero-shot know-it-all personal assistant. Appendix A discusses how to install PyTorch on your computer, with or without a compute unified device architecture-enabled GPU. Appendix B provides information on what background you need in order to proceed with projects in this book and some basic concepts in deep learning such as loss functions, activation functions, and optimizers. ### About the Code This book contains many examples of source code both in numbered listings and in line with normal text. In both cases, source code is formatted in a fixed-width font like this to separate it from ordinary text. Sometimes code is also **in bold** to highlight code that has changed from previous steps in the chapter, such as when a new feature adds to an existing line of code. In many cases, the original source code has been reformatted; we've added line breaks and reworked indentation to accommodate the available page space in the book. In rare cases, even this was not enough, and listings include line-continuation markers (➥). Additionally, comments in the source code have often been removed from the listings when the code is described in the text. Code annotations accompany many of the listings, highlighting important concepts. ### liveBook Discussion Forum