Dynamic Activation Functions in Deep Neural Networks

Activation functions are considered as main component in artificial neural networks. The current paper considers learning activation functions with combination of activation functions. We propose two approaches to use activation functions and construction of adaptive activation parameters to input data. Namely, to show effectiveness, we investigate linear form and non-linear form to combine activation functions, then introduce adaptive activation function. Numerical experiments show the proposed activation techniques overcome by performances and accuracy than standard rectified unit family functions.