Generative Models

1. Supervised vs Unsupervised Learning

2. Taxonomy of Generative Models

3. Fully Visible Belief Network (FVBN)

4. Variational Autoencoders (VAE)

Generative Modeling

• Given training data, generate new samples from same distribution.

• Formulate as density estimation problems:

  1. Explicit density estimation: explicit define and solve pmodel(x)

  2. Implicit density estimation: learn model that can sample from pmodel(x) without explicity defining it.

Taxonomy of Generative Models

Fully Visible Belief Network (FVBN)

• An explicit density model.

• Use Chain rule to decompose likelihood of an image x into product of 1-d distributions.

• Then, maximize likelihood of training data.

• Complex distribution over pixel values ==> Express using a neural network.

Some Explainations of FVBNs: [Link]

• What distinguishes the FVBNs from the other explicit density models are that they have a tractable density: that is, that you can exactly calculate the probability of samples of your dataset (or at least, that's the assumption made by the model). This is what's meant by "fully visible".

Fully Visible Belief Network (FVBN)

• Use Chain rule to decompose likelihood of an image x into product of 1-d distributions.

Pixel RNN vs Pixel CNN

• 2 different approaches to generative modeling of images at the pixel level. Both models aim to generate realistic images pixel by pixel.

  1. Pixel RNN uses recurrent connections and generates pixels sequentially.

  2. Pixel CNN uses convolutional layers to model dependencies and can generate pixels in parallel. 

1. Pros:

   • Can explicitly compute likelihood p(x).

   • Easy to optimize.

   • Good samples.

2. Con:

   • Sequential generation => slow.

Improving PixelCNN performance (Salimans et al. 2017 - PixelCNN++)

1. Gated convolutional layers

2. Short-cut connections

3. Discretized logistic loss

4. Multi-scale

5. Training trick.

Improving PixelCNN performance (Salimans et al. 2017 - PixelCNN++)

1. Gated convolutional layers

2. Short-cut connections

3. Discretized logistic loss

4. Multi-scale

5. Training trick.

Variational Autoencoders (VAE)

• Unsupervised approach for learning a lower-dimensional feature representation from unlabeled training data.

• z usually smaller than x (dimensionality reduction). (Want features to capture meaningful factors of variation in data)

• Learning z? (Train such that features can be used to reconstruct original data “Autoencoding” - encoding input itself)

• After training, throw away decoder.

• Transfer from large, unlabeled dataset to small, labeled dataset.

How do we make autoencoder a generative model?

• Probabilistic spin on autoencoders - will let us sample from the model to generate data!

• z is latent factors used to generate x: attributes, orientation, etc. 

• We want to estimate the true parameters θ* of this generative model given training data x.

How should we represent the model?

• Choose prior p(z) to be simple, e.g. Gaussian. Reasonable for latent attributes, e.g. pose, how much smile.

• Conditional p(x|z) is complex (generates image) => represent with neural network.

How to train the model

• Variational Autoencoders (VAEs) define intractable density function with latent z: 

Intractability