Attention Mechanism Summary

The attention mechanism in Deep Learning is derived from the attention mechanism of the human brain. When the human brain receives external information, such as visual or auditory information, it often does not process and understand a whole scene in its entirety at once, but only focusing attention selectively on some significant or interesting information which help filter out un-important information and improve the efficiency of information processing.

Attention

Attention in Cognitive Neurology

Top-down conscious attention, called Focus Attention.
Bottom-up unconscious attention, called Saliency-Based Attention.

Attention-based on Calculated Area

Soft Attention - Global Attention.
Hard Attention.
Local Attention.

Attention-based on Information Used

General Attention.
Local Attention.

Attention-based on Strature Level

Single-layer Attention.
Multi-layer Attention.
Multi-head Attention.

Model Aspects:

CNN-Attention.
LSTM-Attention.
Pure Attention.

Core idea

Attention = Weight x Value.
weight = Function(Query, Key).
X = [x1, ..., xN] ∈ RDxN.
To find attention for a certain task, it is necessary to calculate the weight of each input vector and decide which one is more important.
Introducing the Query q vector related to the task, it is necessary to calculate the correlation between each input vector and the query vector according to the task.
Use z=n to indicate that the input vector at the n position is selected.

where: αn is called the attention distribution, s(xn, q) is called the attention scoring function. In this way, the attention distribution of each input vector under a specific task is calculated, which is the importance of each input.

Many forms of attention function

Additive model: s(xn, q) = vTtanh(Wx+Uq).
Dot product model: s(xn, q) = xTq.
Scaled dot product model: s(xn, q) = xTq/sqrt(D).
Bilinear model: s(xn, q) = xTWq.

Among them W, U, v are the learnable parameters and D is the dimension of the input vector.

Mechanisms

Soft-attention

Weighted average of input information.

Key-value Pair Attention

Using key-value pairs to represent input information,
1. key to calculate attention distribution
2. value to calculate aggregate information

(K,V) = [(k1, v1),...,(kN, vN)] represents the input information of the N group.

when K=V, the key-value pairs mode degenerates into general mechanisms of attention.

The Multi-Head Attention

Uses multiple queries Q = [q1, ..., qM] to select multiple sets of information from the input information in parallel, and each attention focuses on a different part of the input information.

o: represents vector splicing

Self-Attention

The purpose of the self-attention mechanism is to solve the problem of dynamically generating different connection weights
For self-attention, Wq, Wk, and Wv are learned linear transformations that are shared across all spatial locations, and respectively produce queries, keys, and values when used to transform x.

The input sequence is X = [x1, ..., xN] ∈ RDx*N,
The output sequence is H = [h1, ..., hN] ∈ RDv*N.
1. For the input, first map it linearly to three different spaces to get Q, K, V
2. For each query vector qn ∈ Q, calculate the output vector
αnj indicates the weight of the nth output to the ith input

Application

For unsupervised language model training like GPT, Q,K,V are usually from the same source, so such operation is also called self-attention.
For the machine translation task in the second paper, it first applies self-attention separately to source and target sequences, then on top of that it applies another attention where Q is from the target sequence and K,V are from the source sequence.
For recommendation systems, Q can be from the target items, K,V can be from the user profile and history.

Multi-head Self-attention Model

In order to extract more interactive information, a multi-headed self-attention model can be used to capture different interactive information in multiple different projection spaces.
Assuming that the self-attention model is applied in M projection spaces, there are: