DCANet: Learning Connected Attentions for Convolutional Neural Networks 

0) Motivation, Object and Related works:

Motivation:

• While self-attention mechanism has shown promising results for many vision tasks, it only considers the current features at a time. 

• We show that such a manner cannot take full advantage of the attention mechanism. 

Objectives:

1. Present Deep Connected Attention Network (DCANet), a novel design that boosts attention modules in a CNN model without any modification of the internal structure. 

   1. We interconnect adjacent attention blocks, making information flow among attention blocks possible. 

   2. With DCANet, all attention blocks in a CNN model are trained jointly, which improves the ability of attention learning. 

   3. Our DCANet is generic. It is not limited to a specific attention module or base network architecture. 

2. Experimental results on ImageNet and MS COCO benchmarks show that DCANet consistently outperforms the state-of-the-art attention modules with a minimal additional computational overhead in all test cases. All code and models are made publicly available. 

Related works:

1. Self-attention mechanisms:

   • Explores the interdependence within the input features for a better representation. 

   • Large range of tasks, from machine translation [3] in natural language processing to object detection [7] in computer vision.

1) Channel interdependencies, SENet [18], GENet [17] and SGENet [20] leverage self-attention for contextual modeling. 

2) Global context information, NLNet [38] and GCNet [7] introduce self-attention to capture long-range dependencies in non-local operations. 

3) Consider both channel-wise and spatial attentions: BAM [26] and CBAM [39]. 

4) Beyond channel and spatial dependencies, SKNet [21] applies self-attention to kernel size selection. 

2. Residual connections. 

   • By introducing a shortcut connection, neural networks are decomposed into biased and centered subnets to accelerate gradient descent. 

     1. ResNet [14,15] adds an identity mapping to connect the input and output of each convolutional block, which drastically alleviates the degradation problem [14] and opens up the possibility for deep convolutional neural networks. 

     2. DenseNet [19] connects each block to every other block in a feed-forward fashion. 

     3. FishNet [32] connects layers in pursuit of propagating gradient from deep layers to shallow layers. 

     4. DLA [40] shows that residual connection is a common approach of layer aggregation. By iteratively and hierarchically aggregating layers in a network, DLA is able to reuse feature maps generated by each layer. 

   • They are still fairly new when it comes to integration with attention mechanisms.

     1. RANet [37] utilizes residual connections in attention block; [37,26], residual learning is used in attention modules to facilitate the gradient flow. 

3. Connected Attention. 

   • RA-CNN recurrently generates attention regions based on current prediction to learn the most discriminative region. By doing so, RA-CNN obtains an attention region from coarse to fine. 

   • In GANet [5], the top attention maps generated by customized background attention blocks are up-sampled and sent to bottom background attention blocks to guide attention learning. 

   • Different from the recurrent and feed-backward methods, our DCA module enhances attention blocks in a feed-forward fashion, which is more computation-friendly and easier to implement.