def selu(x): """Scaled Exponential Linear Unit (SELU). The Scaled Exponential Linear Unit (SELU) activation function is: `scale * x` if `x > 0` and `scale * alpha * (exp(x) - 1)` if `x < 0` where `alpha` and `scale` are pre-defined constants (`alpha = 1.67326324` and `scale = 1.05070098`). The SELU activation function multiplies `scale` > 1 with the `[elu](https://www.tensorflow.org/versions/r2.0/api_docs/python/tf/keras/activations/elu)` (Exponential Linear Unit (ELU)) to ensure a slope larger than one for positive net inputs. The values of `alpha` and `scale` are chosen so that the mean and variance of the inputs are preserved between two consecutive layers as long as the weights are initialized correctly (see [`lecun_normal` initialization] (https://www.tensorflow.org/api_docs/python/tf/keras/initializers/lecun_normal)) and the number of inputs is "large enough" (see references for more information). ![]https://cdn-images-1.medium.com/max/1600/1*m0e8lZU_Zrkh4ESfQkY2Pw.png (Courtesy: Blog on Towards DataScience at https://towardsdatascience.com/selu-make-fnns-great-again-snn-8d61526802a9) Example Usage: >>> n_classes = 10 #10-class problem >>> from tensorflow.python.keras.layers import Dense >>> model = tf.keras.Sequential() >>> model.add(Dense(64, kernel_initializer='lecun_normal', ... activation='selu', input_shape=(28, 28, 1))) >>> model.add(Dense(32, kernel_initializer='lecun_normal', ... activation='selu')) >>> model.add(Dense(16, kernel_initializer='lecun_normal', ... activation='selu')) >>> model.add(Dense(n_classes, activation='softmax')) Arguments: x: A tensor or variable to compute the activation function for. Returns: The scaled exponential unit activation: `scale * elu(x, alpha)`. # Note - To be used together with the initialization "[lecun_normal] (https://www.tensorflow.org/api_docs/python/tf/keras/initializers/lecun_normal)". - To be used together with the dropout variant "[AlphaDropout] (https://www.tensorflow.org/api_docs/python/tf/keras/layers/AlphaDropout)". References: [Self-Normalizing Neural Networks (Klambauer et al, 2017)] (https://arxiv.org/abs/1706.02515) """ return nn.selu(x)
def selu(x): """Scaled Exponential Linear Unit (SELU). The Scaled Exponential Linear Unit (SELU) activation function is defined as: - `if x > 0: return scale * x` - `if x < 0: return scale * alpha * (exp(x) - 1)` where `alpha` and `scale` are pre-defined constants (`alpha=1.67326324` and `scale=1.05070098`). Basically, the SELU activation function multiplies `scale` (> 1) with the output of the `tf.keras.activations.elu` function to ensure a slope larger than one for positive inputs. The values of `alpha` and `scale` are chosen so that the mean and variance of the inputs are preserved between two consecutive layers as long as the weights are initialized correctly (see `tf.keras.initializers.LecunNormal` initializer) and the number of input units is "large enough" (see reference paper for more information). Example Usage: >>> num_classes = 10 # 10-class problem >>> model = tf.keras.Sequential() >>> model.add(tf.keras.layers.Dense(64, kernel_initializer='lecun_normal', ... activation='selu')) >>> model.add(tf.keras.layers.Dense(32, kernel_initializer='lecun_normal', ... activation='selu')) >>> model.add(tf.keras.layers.Dense(16, kernel_initializer='lecun_normal', ... activation='selu')) >>> model.add(tf.keras.layers.Dense(num_classes, activation='softmax')) Args: x: A tensor or variable to compute the activation function for. Returns: The scaled exponential unit activation: `scale * elu(x, alpha)`. Notes: - To be used together with the `tf.keras.initializers.LecunNormal` initializer. - To be used together with the dropout variant `tf.keras.layers.AlphaDropout` (not regular dropout). References: - [backendlambauer et al., 2017](https://arxiv.org/abs/1706.02515) """ return nn.selu(x)