Attention Mechanism in CNNs

The attention mechanism re-weights the original feature maps by modeling the dependencies between channels or pixels to filter or enhance the channels or areas of interest, which can effectively improve the quality of features and may benefit small object detection.

Channel attention: (look what) allocates resources on each convolution channel and adjusts the z-axis in a single dimension - Focuses on the relationship between channels.
Spatial attention: (look where) adjusts the attention of each position of the feature map and adjusts (x, y) two-dimensionally to make the model pay more attention to the area that deserves more attention - Focuses on the interrelationships of different areas.
Mixed Attention: The integration of channels and spatial attention.

Dataset

Papers

Paper List

Crowd Computing

[SDANet] Shallow Feature Based Dense Attention Network for Crowd Counting_AAAI2020 [Paper]

Attention

CBAM: Convolutional Block Attention Module_2018 [Paper]
[CCBAM] S Zhao, TH Nguyen, B Ma, "Monaural Speech Enhancement with Complex Convolutional Block Attention Module and Joint Time Frequency Losses", IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2021.
AttaNet: Attention-Augmented Network for Fast and Accurate Scene Parsing_AAAI_2020 [Paper] [Personal Summary]
Rotate to Attend: Convolutional Triplet Attention Module [Paper] [Code]

Non-local and its variant

Dynamic Graph Message Passing Networks (CVPR2020)
Local Relation Networks for Image Recognition (CVPR19)
CARAFE: Content-Aware ReAssembly of FEatures (similar to dynamic filter network) (ICCV19)
A2-Nets: Double Attention Networks (NIPS18)
Graph-Based Global Reasoning Networks (CVPR19)
LatentGNN Learning Efficient Non-local Relations (ICML19)
Adaptive Pyramid Context Network for Semantic Segmentation (CVPR19)
Asymmetric Non-local Neural Networks for Semantic Segmentation (ICCV19)

Attention without Attention modules

LAMBDA NETWORKS: MODELING LONG-RANGE INTERACTIONS WITHOUT ATTENTION_ICLR_2021 [Paper]

Reading List

https://lilianweng.github.io/posts/2018-06-24-attention/#a-family-of-attention-mechanisms (add from this page)

Channel attention

The squeeze and excitation network (SENet) [17] uses the global pooling information of each channel to learn the relationship between channels, and uses compression and expansion methods to model channel relationships thereby increasing the weight of important channels.
Dynamic filter networks (DFN) [18] import information from the high stage to help learn the channel weight of the low stage.
FcaNet: Frequency Channel Attention Networks [Paper]

Spatial attention

Deformable Convolution
Residual Attention Network presents residual attention to learn a soft mask. The response of the background information can be better suppressed after the mask.
Adaptively spatial feature fusion (ASFF) [7] utilizes spatial attention to optimize the feature fusion process.
Non-local neural networks [19] capture long-range dependencies, which perform well in capturing contextual information
FASSD: A Feature Fusion and Spatial Attention-Based Single Shot Detector for Small Object Detection

3D Attention

[VAN] "Visual Attention Network". (Large Kernel Attention - LKA + Channel Attention)
[SegNeXt] "SegNeXt: Rethinking Convolutional Attention Design for Semantic Segmentation" (Multi Scale Conv Attention - MSCA)

The integration of channels and spatial attention:

Global context network (GCNet) [20] combines a simplified non-local (NL) block and an SE module.
The convolutional block attention module (CBAM) [21] learns channel and spatial attention weights from AvgPool and MaxPool values, and the two attention mechanisms are executed serially.
Dual attention networks (DANet) [22] fuse dual attention results by element-wise sum to capture rich contextual dependencies for the scene segmentation task.
Spatial and channel-wise attention in convolutional networks (SCA-CNN) [23] combine channel-wise attention and spatial attention on multi-layer feature maps to perform image captioning.

Application:

COVID-19 Detection from X-ray Images using Multi-Kernel-Size Spatial-Channel Attention Network_PR_2021 [Link]
Align, Attend and Locate: Chest X-ray Diagnosis via Contrast Induced Attention Network with Limited Supervision_ICCV_2019 [Link]
Learning Visual Context by Comparison
Local-to-Global Self-Attention in Vision Transformers_2021
Learning to Segment from Scribbles using Multi-scale Adversarial Attention Gates_TMI_2021 [Paper]
Attention U-Net: Learning Where to Look for the Pancreas [Paper]
AMFFCN: Attentional Multi-layer Feature Fusion Convolution Network for Audio-visual Speech Enhancement [Paper]
SSA-CNN: Semantic Self-Attention CNN for Pedestrian Detection [Paper]
AASeg: Attention Aware Network for Real Time Semantic Segmentation [Paper]

Grad-CAM

Axiom-based Grad-CAM: Towards Accurate Visualization and Explanation of CNNs [Paper]

Crowd Computing: [Link]

[CFANet] Coarse- and Fine-grained Attention Network with Background-aware Loss for Crowd Density Map Estimation (WACV) [paper][code]
[ASNet] Attention Scaling for Crowd Counting (CVPR) [paper] [code]
[CWAN] Weakly Supervised Crowd-Wise Attention For Robust Crowd Counting (ICASSP) [paper]
[AGRD] Attention Guided Region Division for Crowd Counting (ICASSP) [paper]
[BBA-NET] BBA-NET: A Bi-Branch Attention Network For Crowd Counting (ICASSP) [paper]
[ASPDNet] Counting dense objects in remote sensing images (ICASSP) [paper]
[HA-CCN] HA-CCN: Hierarchical Attention-based Crowd Counting Network (TIP) [paper]
[CAT-CNN] Crowd counting with crowd attention convolutional neural network (Neurocomputing) [paper]
[RANet] Relational Attention Network for Crowd Counting (ICCV)[paper]
[ANF] Attentional Neural Fields for Crowd Counting (ICCV)[paper]
[CFF] Counting with Focus for Free (ICCV) [paper][code]
[DADNet] DADNet: Dilated-Attention-Deformable ConvNet for Crowd Counting (ACM MM) [paper]
[RAZ-Net] Recurrent Attentive Zooming for Joint Crowd Counting and Precise Localization (CVPR) [paper]
[ADCrowdNet] ADCrowdNet: An Attention-injective Deformable Convolutional Network for Crowd Understanding (CVPR) [paper]
[IA-DNN] Inverse Attention Guided Deep Crowd Counting Network (AVSS Best Paper) [paper]
[SAAN] Crowd Counting Using Scale-Aware Attention Networks (WACV) [paper]
[MRA-CNN] Multi-resolution attention convolutional neural network for crowd counting (Neurocomputing) [paper]
[ACM-CNN] Attend To Count: Crowd Counting with Adaptive Capacity Multi-scale CNNs (Neurocomputing) [paper]
[SCAR] SCAR: Spatial-/Channel-wise Attention Regression Networks for Crowd Counting (Neurocomputing) [paper][code]
[DecideNet] DecideNet: Counting Varying Density Crowds Through Attention Guided Detection and Density (CVPR) [paper]

AAAI 2020:

Explicitly Modeled Attention Maps for Image Classification
582: Efficient License Plate Recognition via Holistic Position Attention
Split then Refine: Stacked Attention-Guided ResUNets for Blind Single Image Visible Watermark Removal
MANGO: A Mask Attention Guided One-Stage Scene Text Spotter
Self-Supervised Attention-Aware Reinforcement Learning
TDAF: Top-Down Attention Framework for Vision Tasks
Object Relation Attention for Image Paragraph Captioning
DAST: Unsupervised Domain Adaptation in Semantic Segmentation Based on Discriminator Attention and Self-Training
1951: Cascade Network with Guided Loss and Hybrid Attention for Finding Good Correspondences
1975: Dual Sparse Attention Network for Session-Based Recommendation
2832: Regional Attention with Architecture-Rebuilt 3D Network for RGB-D Gesture Recognition
4355: Efficient Folded Attention for Medical Image Reconstruction and Segmentation
5186: Attention-Based Multi-Level Fusion Network for Light Field Depth Estimation
6288: A Supervised Multi-Head Self-Attention Network for Nested Named Entity Recognition
6429: Patch-Wise Attention Network for Monocular Depth Estimation
9885: Context-Aware Attentional Pooling (CAP) for Fine-Grained Visual Classification
HOT-VAE: Learning High-Order Label Correlation for Multi-Label Classification via AttentionBased Variational Autoencoders
2261: Arbitrary Video Style Transfer via Multi-Channel Correlation

To-do List

HAR-Net: Joint Learning of Hybrid Attention for Single-stage Object Detection_2019 [Paper] [Code]
[AttractioNet] Attend Refine Repeat: Active Box Proposal Generation via In-Out Localization_2016 [Paper] [Code]

Unfilter

AdaAttN: Revisit Attention Mechanism in Arbitrary Neural Style Transfer [Paper]
Residual Attention: A Simple but Effective Method for Multi-Label Recognition [Paper]
Looking for the Devil in the Details: Learning Trilinear Attention Sampling Network for Fine-grained Image Recognition_2019 [Paper]
RANet: Ranking Attention Network for Fast Video Object Segmentation_2019 [Paper]
Multi-image Super Resolution of Remotely Sensed Images using Residual Feature Attention Deep Neural Networks [Paper]
LBAM: a learnable two-way attention mechanism for image restoration [Paper]
Causal Attention for Unbiased Visual Recognition [Paper]
Residual Attention: A Simple but Effective Method for Multi-Label Recognition [Paper]
Lightweight multi-scale residual networks with attention for image super-resolution [Paper]
Attention-Mechanism-Containing Neural Networks for High-Resolution Remote Sensing Image Classification_2018 [Paper]
DCANet: Learning Connected Attentions for Convolutional Neural Networks [Paper]
[SGE] Spatial Group-wise Enhance: Improving Semantic Feature Learning in Convolutional Networks [Paper]
GCNet: Non-local Networks Meet Squeeze-Excitation Networks and Beyond [Paper]
[NL-SAM] Nonlocal spatial attention module for image classification [Paper]
Interaction-aware Spatio-temporal Pyramid Attention Networks for Action Classification [Paper]
Residual Attention: A Simple But Effective Method for Multi-Label Recognition [Paper] [Code]
[To Tony] ACTION-Net: Multipath Excitation for Action Recognition [Paper] [Code]
FcaNet: Frequency Channel Attention Networks [Paper] [Code]
AFF: Attention feature fusion
SGENet: Spatial Group-wise Enhance: Enhancing Semantic Feature Learning in Convolutional Networks [Paper]
SRMNet: SRM: A Style-based Recalibration Module for Convolutional Neural Networks [Paper]
Polarized Self-Attention: Towards High-quality Pixel-wise Regression [Paper]

Notes

Capture long-range dependencies
- Long-short term memory-networks for machine reading_arXiv_2016
- Attention is all you need_NIPs_2017
- OCNet: Object Context Network for Scene Parsing_arXiv_2018 [Paper]
- Dual attention network for scene segmentation_CVPR_2019
Capture non-local contextual information with limited computation
- Nonlocal neural networks_CVPR_2018
Model spatial and channel relationships
- A relation-augmented fully convolutional network for semantic segmentation in aerial scenes_CVPR_2019
Leverages two criss-cross attention modules
- Ccnet: Criss-cross attention for semantic segmentation_ICCV_2019 [Paper]
Directly exploits class-level context
- Acfnet: Attentional class feature network for semantic segmentation_ICCV_2019
Strengthen the contextual consistency in a specific direction
- AttaNet: Attention-Augmented Network for Fast and Accurate Scene Parsing_AAAI_2020
Use image structure to help handcraft relevant spatial patterns to attend
- [18] Huiyu Wang, Y. Zhu, B. Green, H. Adam, A. Yuille, and Liang-Chieh Chen. Axial-deeplab: Stand-alone axial-attention for panoptic segmentation. ArXiv, abs/2003.07853, 2020.
- [19] Zilong Huang, Xinggang Wang, Lichao Huang, C. Huang, Yunchao Wei, and Wenyu Liu. Ccnet: Criss-cross attention for semantic segmentation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 603–612, 2019.
- [20] Zhen Zhu, Mengdu Xu, Song Bai, Tengteng Huang, and X. Bai. Asymmetric non-local neural networks for semantic segmentation. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pages 593–602, 2019.
- [21] Lang Huang, Y. Yuan, Jianyuan Guo, Chao Zhang, X. Chen, and Jingdong Wang. Interlaced sparse self-attention for semantic segmentation. ArXiv, abs/1907.12273, 2019.
- [22] Prajit Ramachandran, Niki Parmar, Ashish Vaswani, I. Bello, Anselm Levskaya, and Jonathon Shlens. Stand-alone self-attention in vision models. ArXiv, abs/1906.05909, 2019.

References

https://github.com/gjy3035/Awesome-Crowd-Counting/blob/master/src/Attention.md
Pytorch Lib: https://github.com/implus/PytorchInsight
Attention in Neural Networks: https://towardsdatascience.com/attention-in-neural-networks-e66920838742

Page updated

Google Sites

Report abuse