Activation Functions

The activation function in the neural network is to increase non-linearity. The activation function selects a threshold, that is, when it is greater than a certain value, it will be activated, and it will output 0 when it is less than or equal to.

In fact, doing so is in line with human intuition. For brain cells, there should also be a certain threshold, and the cell will be activated or become sensitive. Because cells or brain nerves have different thresholds for external stimuli (or different response values), different people have different sensitivity to the same stimulus.

The activation function can also be understood as an affine transformation (spatial mapping) of the feature space, in which a linear boundary is found. The physical structure of a typical neuron includes a cell body, axons that send signals to other neurons, and dendrites that receive signals or information from other neurons. Neurons receive signals from other neurons through dendrites. The weight of the dendrites is called the synaptic weight, which is multiplied by the received signal. The signal from the dendrites accumulates continuously in the cell body. If the signal intensity exceeds a certain threshold, the neuron transmits information to the axon. If it does not exceed, the signal is "killed" by the neuron and cannot be propagated further.

Dataset

Papers

[ReLU] Vinod Nair and Geoffrey E Hinton. "Rectified linear units improve restricted boltzmann machines". In Icml, 2010.
[LReLU] Andrew L Maas, Awni Y Hannun, Andrew Y Ng, et al. "Rectifier nonlinearities improve neural network acoustic models". In Proc. icml, volume 30, page 3. Citeseer, 2013.
[Swish] Prajit Ramachandran, Barret Zoph, and Quoc V Le. "Searching for activation functions". arXiv preprint arXiv:1710.05941, 2017.
[SiLU] [dSiLU] Stefan Elfwing, Eiji Uchibe, and Kenji Doya. "Sigmoid-weighted linear units for neural network function approximation in reinforcement learning". Neural Networks, 107:3–11, 2018.
[Mish] Diganta Misra. "Mish: A self regularized non-monotonic neural activation function". arXiv preprint arXiv:1908.08681, 4(2):10–48550, 2019.

Reading List

Softmax
LogSoftmax
Modified softmax-A-softmax
L-softmax
ReLu solves the gradient vanish problem.
LReLU and PReLU: solve the problem that the gradient of ReLU is zero when the output is less than zero.
ReLU6 and hard-Swish: specially designed for quantization networks.
SELU: self-normalize a neural network.
Swish and Mish: continuously differentiable activation function.

To-do List

Sigmoid
Tanh
ACON
Hybrid DNN

Activation Functions

Dataset

Papers

Reading List

To-do List

References

About Me: