GAN Hacks to Train Stable GAN

Generative Adversarial Networks, or GANs, are challenging to train.

This is because the architecture involves both a generator and a discriminator model that compete in a zero-sum game. It means that improvements to one model come at the cost of a degrading of performance in the other model. The result is a very unstable training process that can often lead to failure, e.g. a generator that generates the same image all the time or generates nonsense.

As such, there are a number of heuristics or best practices (called “GAN hacks“) that can be used when configuring and training your GAN models. These heuristics are been hard won by practitioners testing and evaluating hundreds or thousands of combinations of configuration operations on a range of problems over many years.

Some of these heuristics can be challenging to implement, especially for beginners.

Further, some or all of them may be required for a given project, although it may not be clear which subset of heuristics should be adopted, requiring experimentation. This means a practitioner must be ready to implement a given heuristic with little notice.

In this tutorial, you will discover how to implement a suite of best practices or GAN hacks that you can copy-and-paste directly into your GAN project.

After reading this tutorial, you will know:

• The best sources for practical heuristics or hacks when developing generative adversarial networks.
• How to implement seven best practices for the deep convolutional GAN model architecture from scratch.
• How to implement four additional best practices from Soumith Chintala’s GAN Hacks presentation and list.

Let’s get started.

Tutorial Overview

This tutorial is divided into three parts; they are:

1. Heuristics for Training Stable GANs
2. Best Practices for Deep Convolutional GANs
   1. Downsample Using Strided Convolutions
   2. Upsample Using Strided Convolutions
   3. Use LeakyReLU
   4. Use Batch Normalization
   5. Use Gaussian Weight Initialization
   6. Use Adam Stochastic Gradient Descent
   7. Scale Images to the Range [-1,1]
3. Soumith Chintala’s GAN Hacks
   1. Use a Gaussian Latent Space
   2. Separate Batches of Real and Fake Images
   3. Use Label Smoothing
   4. Use Noisy Labels

Heuristics for Training Stable GANs

GANs are difficult to train.

At the time of writing, there is no good theoretical foundation as to how to design and train GAN models, but there is established literature of heuristics, or “hacks,” that have been empirically demonstrated to work well in practice.

As such, there are a range of best practices to consider and implement when developing a GAN model.

Perhaps the two most important sources of suggested configuration and training parameters are:

1. Alec Radford, et al’s 2015 paper that introduced the DCGAN architecture.
2. Soumith Chintala’s 2016 presentation and associated “GAN Hacks” list.

In this tutorial, we will explore how to implement the most important best practices from these two sources.

Best Practices for Deep Convolutional GANs

Perhaps one of the most important steps forward in the design and training of stable GAN models was the 2015 paper by Alec Radford, et al. titled “Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks.”

In the paper, they describe the Deep Convolutional GAN, or DCGAN, approach to GAN development that has become the de facto standard.

We will look at how to implement seven best practices for the DCGAN model architecture in this section.

1. Downsample Using Strided Convolutions

The discriminator model is a standard convolutional neural network model that takes an image as input and must output a binary classification as to whether it is real or fake.

It is standard practice with deep convolutional networks to use pooling layers to downsample the input and feature maps with the depth of the network.

This is not recommended for the DCGAN, and instead, they recommend downsampling using strided convolutions.

This involves defining a convolutional layer as per normal, but instead of using the default two-dimensional stride of (1,1) to change it to (2,2). This has the effect of downsampling the input, specifically halving the width and height of the input, resulting in output feature maps with one quarter the area.

The example below demonstrates this with a single hidden convolutional layer that uses downsampling strided convolutions by setting the ‘strides‘ argument to (2,2). The effect is the model will downsample the input from 64×64 to 32×32.

Running the example shows the shape of the output of the convolutional layer, where the feature maps have one quarter of the area.

2. Upsample Using Strided Convolutions

The generator model must generate an output image given as input at a random point from the latent space.

The recommended approach for achieving this is to use a transpose convolutional layer with a strided convolution. This is a special type of layer that performs the convolution operation in reverse. Intuitively, this means that setting a stride of 2×2 will have the opposite effect, upsampling the input instead of downsampling it in the case of a normal convolutional layer.

By stacking a transpose convolutional layer with strided convolutions, the generator model is able to scale a given input to the desired output dimensions.

The example below demonstrates this with a single hidden transpose convolutional layer that uses upsampling strided convolutions by setting the ‘strides‘ argument to (2,2).

The effect is the model will upsample the input from 64×64 to 128×128.

Running the example shows the shape of the output of the convolutional layer, where the feature maps have quadruple the area.

3. Use LeakyReLU

The rectified linear activation unit, or ReLU for short, is a simple calculation that returns the value provided as input directly, or the value 0.0 if the input is 0.0 or less.

It has become a best practice when developing deep convolutional neural networks generally.

The best practice for GANs is to use a variation of the ReLU that allows some values less than zero and learns where the cut-off should be in each node. This is called the leaky rectified linear activation unit, or LeakyReLU for short.

A negative slope can be specified for the LeakyReLU and the default value of 0.2 is recommended.

Originally, ReLU was recommend for use in the generator model and LeakyReLU was recommended for use in the discriminator model, although more recently, the LeakyReLU is recommended in both models.

The example below demonstrates using the LeakyReLU with the default slope of 0.2 after a convolutional layer in a discriminator model.

Running the example demonstrates the structure of the model with a single convolutional layer followed by the activation layer.

4. Use Batch Normalization

Batch normalization standardizes the activations from a prior layer to have a zero mean and unit variance. This has the effect of stabilizing the training process.

Batch normalization is used after the activation of convolution and transpose convolutional layers in the discriminator and generator models respectively.

It is added to the model after the hidden layer, but before the activation, such as LeakyReLU.

The example below demonstrates adding a Batch Normalization layer after a Conv2D layer in a discriminator model but before the activation.

Running the example shows the desired usage of batch norm between the outputs of the convolutional layer and the activation function.

5. Use Gaussian Weight Initialization

Before a neural network can be trained, the model weights (parameters) must be initialized to small random variables.

The best practice for DCAGAN models reported in the paper is to initialize all weights using a zero-centered Gaussian distribution (the normal or bell-shaped distribution) with a standard deviation of 0.02.

The example below demonstrates defining a random Gaussian weight initializer with a mean of 0 and a standard deviation of 0.02 for use in a transpose convolutional layer in a generator model.

The same weight initializer instance could be used for each layer in a given model.

6. Use Adam Stochastic Gradient Descent

Stochastic gradient descent, or SGD for short, is the standard algorithm used to optimize the weights of convolutional neural network models.

There are many variants of the training algorithm. The best practice for training DCGAN models is to use the Adam version of stochastic gradient descent with the learning rate of 0.0002 and the beta1 momentum value of 0.5 instead of the default of 0.9.

The Adam optimization algorithm with this configuration is recommended when both optimizing the discriminator and generator models.

The example below demonstrates configuring the Adam stochastic gradient descent optimization algorithm for training a discriminator model.

7. Scale Images to the Range [-1,1]

It is recommended to use the hyperbolic tangent activation function as the output from the generator model.

As such, it is also recommended that real images used to train the discriminator are scaled so that their pixel values are in the range [-1,1]. This is so that the discriminator will always receive images as input, real and fake, that have pixel values in the same range.

Typically, image data is loaded as a NumPy array such that pixel values are 8-bit unsigned integer (uint8) values in the range [0, 255].

First, the array must be converted to floating point values, then rescaled to the required range.

The example below provides a function that will appropriately scale a NumPy array of loaded image data to the required range of [-1,1].

Soumith Chintala’s GAN Hacks

Soumith Chintala, one of the co-authors of the DCGAN paper, made a presentation at NIPS 2016 titled “How to Train a GAN?” summarizing many tips and tricks.

The video is available on YouTube and is highly recommended. A summary of the tips is also available as a GitHub repository titled “How to Train a GAN? Tips and tricks to make GANs work.”

The tips draw upon the suggestions from the DCGAN paper as well as elsewhere.

In this section, we will review how to implement four additional GAN best practices not covered in the previous section.

1. Use a Gaussian Latent Space

The latent space defines the shape and distribution of the input to the generator model used to generate new images.

The DCGAN recommends sampling from a uniform distribution, meaning that the shape of the latent space is a hypercube.

The more recent best practice is to sample from a standard Gaussian distribution, meaning that the shape of the latent space is a hypersphere, with a mean of zero and a standard deviation of one.

The example below demonstrates how to generate 500 random Gaussian points from a 100-dimensional latent space that can be used as input to a generator model; each point could be used to generate an image.

Running the example summarizes the generation of 500 points, each comprised of 100 random Gaussian values with a mean close to zero and a standard deviation close to 1, e.g. a standard Gaussian distribution.

2. Separate Batches of Real and Fake Images

The discriminator model is trained using stochastic gradient descent with mini-batches.

The best practice is to update the discriminator with separate batches of real and fake images rather than combining real and fake images into a single batch.

This can be achieved by updating the model weights for the discriminator model with two separate calls to the train_on_batch() function.

The code snippet below demonstrates how you can do this within the inner loop of code when training your discriminator model.

3. Use Label Smoothing

It is common to use the class label 1 to represent real images and class label 0 to represent fake images when training the discriminator model.

These are called hard labels, as the label values are precise or crisp.

It is a good practice to use soft labels, such as values slightly more or less than 1.0 or slightly more than 0.0 for real and fake images respectively, where the variation for each image is random.

This is often referred to as label smoothing and can have a regularizing effect when training the model.

The example below demonstrates defining 1,000 labels for the positive class (class=1) and smoothing the label values uniformly into the range [0.7,1.2] as recommended.

Running the example summarizes the min and max values for the smooth values, showing they are close to the expected values.

There have been some suggestions that only positive-class label smoothing is required and to values less than 1.0. Nevertheless, you can also smooth negative class labels.

The example below demonstrates generating 1,000 labels for the negative class (class=0) and smoothing the label values uniformly into the range [0.0, 0.3] as recommended.

4. Use Noisy Labels

The labels used when training the discriminator model are always correct.

This means that fake images are always labeled with class 0 and real images are always labeled with class 1.

It is recommended to introduce some errors to these labels where some fake images are marked as real, and some real images are marked as fake.

If you are using separate batches to update the discriminator for real and fake images, this may mean randomly adding some fake images to the batch of real images, or randomly adding some real images to the batch of fake images.

If you are updating the discriminator with a combined batch of real and fake images, then this may involve randomly flipping the labels on some images.

The example below demonstrates this by creating 1,000 samples of real (class=1) labels and flipping them with a 5% probability, then doing the same with 1,000 samples of fake (class=0) labels.

Try running the example a few times.

The results show that approximately 50 “1”s are flipped to 1s for the positive labels (e.g. 5% of 1,0000) and approximately 50 “0”s are flopped to 1s in for the negative labels.

Further Reading

This section provides more resources on the topic if you are looking to go deeper.

Papers

API

Articles

Summary

In this tutorial, you discovered how to implement a suite of best practices or GAN hacks that you can copy-and-paste directly into your GAN project.

Specifically, you learned:

• The best sources for practical heuristics or hacks when developing generative adversarial networks.
• How to implement seven best practices for the deep convolutional GAN model architecture from scratch.
• How to implement four additional best practices from Soumith Chintala’s GAN Hacks presentation and list.