Generative Adversarial Networks

Look at the three faces above. You might notice the similarity in their ages, their expressions, the subtle congruence in skin tone and eye colour. They all gaze straight at you with an endearing smile and an approachable appearance. But there is one thing you would almost certainly never guess about all three of them: they do not exist. Not one of them.

All three individuals were produced by a website called ThisPersonDoesNotExist.com, created by former Uber software engineer Phillip Wang. The site is powered by a type of machine learning framework known as a generative adversarial network, or GAN — a concept first developed by Ian Goodfellow and his colleagues in June 2014. Every face it generates belongs to nobody. Every smile has never been smiled.

The formulation and development of GANs take on strong roots from game theory, which is the mathematical and often computational study of strategic interdependence, where assumed “rational” players make their decisions with the knowledge that their outcomes depend on the actions of others. It is often applied in economics, computer science and political science, and analyses the rationality of the agent’s decisions as they consider the strategies of their opponents to maximise their own chance of success.

At the heart of a GAN is a zero-sum game, which is a concept from game theory which in particular one party's gain is precisely the other's loss, as elaborated upon previously. This game is played out between two competing neural networks: a generator and a discriminator. The generator's sole objective is to produce an image so convincing that it cannot be told apart from a real photograph. The discriminator's objective is the opposite: to catch it out and determine which photo out of the collection it is presented with has been artificially generated.

Essentially, the two networks train against each other in a continuous loop. Each time the discriminator successfully identifies a generated image as fake, it sends that judgement back as a correction, which is a signal that tells the generator exactly where it fell short. The generator, notably, never sees real images directly. It learns entirely from these judgements, adjusting itself each round in pursuit of a single goal: to push the discriminator to the point of uncertainty, where it can do no better than guess. Those three faces above are the product of that process playing out across millions of iterations. What began as noise gradually resolved, through failure and correction, into something the discriminator could no longer reliably distinguish from reality. The generator had won and, in doing so, had learned to construct a human face from nothing at all.

The mathematical approach behind the development of this technique is a lot more complex. The original GAN is defined as the following game:

Ω represents the the set of all possible real images, which essentially entails any photograph that could ever theoretically exist, real or fake. μref is the reference distribution; it represents the pattern of what real and true images actually look like, such as what is present within the image. In our case it would represent what constitutes a real human face, such as having two eyes, ears and nostrils; one pair of lips and one nose, and more unique facial characteristics.

P(Ω) represents the set of all possible probability distributions of images, which μref is a category of. This should not be confused with the axiom of probability; P(Ω) is the set of distributions and does not equal 1. Instead, it denotes the set of every conceivable way you could describe the randomness of the image space. For example, if Ω was the set {heads, tails}, P(Ω) would be the set of all the ways you could assign probabilities to heads and tails:

• {H: 0.5, T: 0.5} – A fair coin
• {H: 0.9, T: 0.1} – Biased in favour of heads
• {H: 1, T: 0} – Always heads

Every one of these is a member of P(Ω), and the set would therefore contain infinitely many elements, since the probability of heads against tails could be any real number between 0 and 1. In the case of our game, imagine that Ω is the set of all possible 256x256 images. P(Ω) would therefore include:

• The distribution that only generates cat photos
• The distribution that only generates photos of human face (what our GAN is attempting to imitate)
• The distribution of every real photo ever taken
• The generator’s current distribution Think of it as the space of all the answers; the generator’s duty in this case is to find the answer that matches closest to μref.

μ_D : Ω → 𝒫[0, 1], where 𝒫[0, 1]

The generator starts somewhere random in P(Ω) producing noise, which would be but will almost certainly never be the distribution it needs to imitate, and training moves it closer through that space between its current distribution and μref. As for the discriminator, its strategy set takes any image from Ω and outputs a probability, which is its confidence between 0 and 1 that the image is real. The official name for this function is a Markov Kernel – when a function maps something to a probability distribution. Put simply, the discriminator is estimating how real the image is.

$$L(\mu_G,\mu_D) := \mathbb{E}_{x\sim\mu_{\mathrm{ref}},\,y\sim\mu_D(x)}[\ln y] + \mathbb{E}_{x\sim\mu_G,\,y\sim\mu_D(x)}[\ln(1-y)]$$

1 represents certainty in the realness of the image and 0 represents the certainty that the image is counterfeit. This is the core of the zero-sum game. The first term, E [ln y], is when the discriminator is presented a real image and outputs y, which is its realness score. If it correctly says y = 1, ln (1) = 0 and it receives no penalty. If it wrongly assumes y = (0.1), ln (0.1) is a large negative number, and the discriminator receives a large penalty. Therefore, it aims to produce a term as close to zero as possible and so wants its y output to be as high as possible.

The second term, E [ln (1 - y)], is where the image produced by the generator is presented to the discriminator. If it correctly says y = 0, then 1 – 0 = 1 and ln(1) = 0, so it receives no penalty. However, if it is deceived by the generator and y is outputted as 1, then 1 – 1 = 0 and ln(0) spirals into negative infinity, and the discriminator is heavily punished. The generator wants y to be as close to 1 as possible to drag the value down, and the discriminator wants y to be as close to 0 as possible to keep the value high, hence the term minimax – the generator attempts to minimise, the discriminator aims to maximise.

The two are on opposite sides of the same coin. The game reaches equilibrium when the generator produces an image so good that the discriminator is forced to produce y = 0.5 for all images it is fed. What you witnessed at the beginning was the product of a perfect zero-sum game in the generator’s favour; three completely indistinguishable human faces that fooled all of you at first glance.