Basics
Activation functions
| | output | | | |
|—————————-|——————|—————-|———————————————————————|——————————————————————————————|
| linear/I | +-infinity | differentiable | 
| cause a neural network to get stuck at the training time. |
| Sigmoid/Logistic | (0 to 1) | differentiable | 
| |
| Tanh/hyperbolic tangent | (-1 to 1) | differentiable | | mapped strongly negative and the zero inputs will be mapped near zero in the tanh graph. |
| ReLU/Rectified Linear Unit | [ 0 to infinity) | | 
| not mapping the negative values appropriately. |
| Leaky ReLU | +-infinity | | | attempt to solve the dying ReLU problem |