Generative Adversarial Networks

What are GANs?

Generative Adversarial Networks (GANs) consist of two neural networks that are competing against each other. One neural network, the “generator” takes a random noise vector to produce fake images. The other network, the “discriminator” is fed real images, and uses those to determine if the fake images made by the generator are real or fake. The generator relies on feedback from the discriminator to get better at creating images, while the discriminator gets better at classifying between real and fake images. In our implementation, our generator and discriminator will be convolutional neural networks.

A popular metaphor is an art forger and art critic:

More specifically, the discriminator is fed (real) training images and learns to distinguish between what's real and what's fake. The generator is given a z-dimensional random noise vector and uses that to generate an image. This picture is then sent to the discriminator to return a value 1 (if real) or 0 (if fake). Once compared against the labels (ground truth: real or fake), the losses are computed. The generator and discriminator are then updated; thus over time, both get better at their jobs.

It's important that discriminator doesn't start off too strong, otherwise the generator won't get any information on how to improve itself. The generator doesn't know what the real data looks like; it relies on the discriminator's feedback. Only when the discriminator and generator are balanced do both improve, and hence lead to the generator producing realistic images.

Architecture of our DCGAN

The model used for this project was coded using Python, TensorFlow, and Nvidia’s CUDA toolkit. Due to the computational power required to train a GAN, we ran instances of our model using GPUs - both locally (Nvidia RTX 2060) and using Google Cloud Platform (Nvidia Tesla K80).

Much of our code was based on this notebook by simoninithomas. A corresponding tutorial, which goes more in depth about the code, is also available here. GANs are difficult to train and sensitive to hyperparameters (you don’t want the generator or discriminator overpowering each other). We tried the hyperparameters used in this notebook, as well as our own. The hyperparameters we tuned were: generator learning rate, discriminator learning rate, batch size, alpha (leak parameter for leaky ReLU), and beta (exponential decay rate for 1st moment in the optimizer). For batch size, it was required to use 32 when trained locally to avoid OOM errors.
The number of epochs run depended on how long we let the model train for, and whether we thought (based on the outputted images) the model was improving/getting worse. For more specific hyperparameters, please see our results below.

Our GAN consisted of two convolutional neural networks pitted against each other. The architecture used for this project is as follows:

From these "GAN hacks" it is recommended that:

• Batch normalization occurs at each layer (except for first layer of generator)
• LeakyReLU as the activation function for both generator and discriminator
• Use Adam optimizer
• tanh as last layer of generator output

Results

We trained multiple GANs on different datasets, and the categories that we're satisified with the results are listed below. The code used resizes images to 128x128 and generates 128x128 sized images (may appear smaller on the website here). We modified the code as to save a version of the model every 50 epochs and generate a 3x3 grid of images after each epoch. Our training process, as well as a curated collection of our best generated images, are below.

Other versions of GANs

In addition to DCGANs, there are also various other implementations of GANs. For experimentation purposes, we also tried using a Wasserstein GAN (WGAN) on the same sunsets dataset to compare the results. The code we used is from goldesel23.

Based on the above results, we believe that our sunset DCGAN lead to better results. The WGAN is supposedly more stable and fixes problems like mode collapse, however a paper from Google Brain at NIPS 2018 found no evidence that other versions of GANs, including WGAN, consistently performed better than the original GAN.

Discussion, tuning hyperparameters, and other datasets

Discussion

How to best tune a GAN is an open research problem. Grid and random search for trying different hyperparmeters are not the best methods due to the computational power and time required to train a GAN. While doing preliminary research for this project, we were worried about tackling GANs because of the time required to train. Mostly, we only found unsure estimates for how long it took to train – hence, when showing our results, we include our actual training times to provide an estimate for what to expect. Additionally, we didn't find many resources for what didn't work, we mostly saw the final result but didn't see the process of getting there. Thus, we are including the below sections to talk about our entire process of working with GANs.
Overall, we noticed that larger datasets do make a difference – our faces dataset was the largest, and the generated images turned out the clearest as well.

Tuning hyperparameters

Dataset: Same 1,487 flower paintings as used above

Model 3 is the model used in the above flower results section. Other models with different hyperparameters, were run and led to worse results.
The results from Model 1 at epochs 300/350 look similar to the results at epoch 500 in Model 3. However, we noticed that the flowers generated using epochs 300 and 350 with these hyperparameters resulted in a lesser quantity of images considered "good" and with less defined features. By letting it run to 500 epochs we experienced mode collapse (when all the generated images start to look the same - see epoch 500). As the generated images don't necessarily get better by training longer, we recommend saving multiple versions of the model during training.
Model 2 led to mode collapse quickly, which is likely due to the tuning of alpha and beta.

Working with other datasets