A Mini-Review on Current Difficulties of Self-attention Generative Adversarial Networks

Generative model and discriminative model are two basic models in machine learning, but the research of generative model is much lower than that of discriminative model, the reason is that the application scene of discriminative model is more abundant, and its effect can be achieved immediately. The research on Generative Adversarial Networks (GAN), one of the most mainstream generative models, has exploded exponentially since it was first brought into the public domain by Ian Goodfellow et al. [1] in the year 2014. The advantage is that there is no need to employ any Markov chains or unrolled approximate inference networks in training and generation process, so it is less expensive to generate a clear image than other methods. The performance of the original GAN was not good enough that the generated content was blurry, but it can be regarded as an alternative generative model in addition to Variational Auto-encoder (VAE) [2,3], PixelCNN [4] and so on.

Since the self-attention mechanism was proposed by [5], it has shone brilliantly in the field of Natural Language Processing (NLP) in the form of Transformer. The solutions, based on autoregressive language modeling in GPT [6-8] and masked autoencoding in BERT [9]. In the field of vision, which is unified by convolutional neural networks [10-13], inspired by the robust development of selfattention in NLP, there is also a lot of work to introduce self-attention to visual tasks [14-17], but the work that makes Transformer favored in the field of vision is [18, 19], the performance of Vision Transformer (ViT) model is better than that of state-of-the-art convolution neural network in some cases [20-22].

Generative adversarial networks

Generative Adversarial Networks were developed by Ian Goodfellow, et al. [1] in the year 2014. GANs belong to the class of Generative models [23]. GANs are based on the min-max, zerosum game theory. For this, GANs consist of two neural networks: one is the Generator and the other is the Discriminator. The goal of the Generator is to learn to generate fake sample distribution to deceive the Discriminator whereas the goal of the Discriminator is to learn to distinguish between real and fake distribution generated by the Generator. The Generator tries to minimize the following function while the Discriminator tries to maximize it. The Minimax loss is given as,

Here, 𝔼_x is the expected value over all real data samples, D(x) is the probability estimate of the Discriminator if x is real, 𝐺(𝑧) is the output of the Generator for a given random noise vector 𝑧 as input, 𝐷(𝐺(𝑧)) is the Discriminator’s probability estimate if the fake generated sample is real, 𝔼_𝑧 is the expected value over all random inputs to the Generator.

Aided by the rapid and numerous follow-up related research, the potential of GANs were gradually released. Some of the more prominent work, such as Conditional GAN (CGAN) [24], GAN just generates samples randomly, and CGAN makes it possible to generate content according to some fixed directories; Wasserstein GAN (WGAN) [25], WGAN use their delicate mathematical derivation to improved stability of training process and get rid of several problems, e.g., mode collapse. In most cases, the purpose of people using GANs is to generate near-real data, such as text and images. In addition, a lot of exciting work has been born in recent years [26-31].

Visual transformer

The original transformer was built for NLP [5], where the multi-head self-attention and feedforward MLP layer are stacked to capture the long-term correlation between words. Its popularity among computer vision tasks rises recently [32-34]. The core of a transformer is the self-attention mechanism, which characterizes the dependencies between any two distant tokens. With the first pure Transformer-based visual backbone network ViT proposed by [18], the number of studies on A has increased significantly. The main reason is that it does not use the convolution structure, but achieved the performance beyond the traditional convolution neural network, and compared with the convolution neural network, it has less computational cost in the training of the model with the same parameters. However, it is also shown in the original paper that this Transformer-based visual model requires a huge amount of data to be fed in order to achieve excellent performance. In the case of less training data, the result is not as good as the traditional convolution neural network, e.g., [35,36].

Evaluation of transformer-based GAN

One of the most difficult aspects of GAN training is assessing their performance, or determining how well a model approximates a data distribution. In terms of theory and applications, significant progress has been made, with a large number of GAN variants now available. However there has been relatively little effort put into evaluating GANs, and there are still gaps in quantitative assessment methods. From the extensive work on evaluation of GANs [37,38], the general evaluation indicators of GANs are the use of Fre chet Inception Distance (FID [39]) and Inception Score (IS [40]). In order to understand the performance of the model more intuitively, some benchmark data sets are usually used for experiments. The most common are CIFAR-10 [41], CelebA [42], ImageNet [43], etc.

Along with the rapid development of Transformer, some work has been done to introduce this self-attention mechanism into the architecture of GAN, and achieved state-of-the-art results. [20,44,45]. Using Transformer to build a GAN model in addition to the advantages of using numerous tricks in the NLP domain, etc., more current difficulties can be summarized as follows:
1. Judging from the latest work, Transformer-based GAN needs to have a deeper understanding of semantic information [44-47].
2. Transformer-based GAN requires stronger attention mechanisms, especially those tailored to the characteristics of generative models, not only performance, but also efficiency, e.g. [48-50].
3. Introduce pre-training into the training of Transformer-based GAN, e.g. [51].
4. Conditional image generation in Transformer-based GAN [26,52].
5. The lack of diversity is a problem in Transformer-based GAN [53-58].

None.

The author declares no conflict of interest.