Beyond Self-attention: External Attention using Two Linear Layers for Visual Tasks

Related Works:

1. The attention mechanism in visual tasks 

• The attention mechanism can be viewed as a mechanism for reallocating resources according to the importance of activation. It plays an important role in the human visual system. There has been vigorous development of this field in the last decade [3], [13], [14], [15], [16], [17], [18]. Hu et al. proposed SENet [15], showing that the attention mechanism can reduce noise and improve classification performance. Subsequently, many other papers have applied it to visual tasks. Wang et al. presented non-local networks [3] for video understanding, Hu et al. [19] used attention in object detection, Fu et al. proposed DANet [4] for semantic segmentation, Zhang et al. [11] demonstrated the effectiveness of the attention mechanism in image generation, and Xie et al. proposed A-SCN [20] for point cloud processing. 

2. Self-attention in visual tasks 

   • Self-attention is a special case of attention, and many papers [3], [4], [11], [17], [21], have considered the self-attention mechanism for vision. The core idea of self-attention is calculating the affinity between features to capture longrange dependencies. However, as the size of the feature map increases, the computing and memory overheads increase quadratically. To reduce computational and memory costs, Huang et al. [5] proposed criss-cross attention, which considers row attention and column attention in turn to capture the global context. Li et al. [6] adopted expectation maximization (EM) clustering to optimize self-attention. Yuan et al. [7] proposed use of object-contextual vectors to process attention; however, it depends on semantic labels. Geng et al. [8] show that matrix decomposition is a better way to model the global context in semantic segmentation and image generation. Other works [22], [23] also explore extracting local information by using the self-attention mechanism. Unlike self-attention which obtains an attention map by computing affinities between self queries and self keys, our external attention computes the relation between self queries and a much smaller learnable key memory, which captures the global context of the dataset. External attention does not rely on semantic information and can be optimized by the back-propagation algorithm in an end-to-end way instead of requiring an iterative algorithm. 

3. Transformer in visual tasks 

   • Transformer-based models have had great success in natural language processing [1], [2], [16], [24], [25], [26], [27]. Recently, they have also demonstrated huge potential for visual tasks. Carion et al. [28] presented an end-to-end detection transformer that takes CNN features as input and generates bounding boxes with a transformer. Dosovitskiy [18] proposed ViT, based on patch encoding and a transformer, showing that with sufficient training data, a transformer provides better performance than a traditional CNN. Chen et al. [29] proposed iGPT for image generation based on use of a transformer. Subsequently, transformer methods have been successfully applied to many visual tasks, including image classification [12], [30], [31], [32], object detection [33], lowerlevel vision [34], semantic segmentation [35], tracking [36], video instance segmentation [37], image generation [38], multimodal learning [39], object re-identification [40], image captioning [41], point cloud learning [42] and self-supervised learning [43]. Readers are referred to recent surveys [44], [45] for a more comprehensive review of the use of transformer methods for visual tasks.