8.2 Activation Functions and Backpropagation

Neural networks are the backbone of deep learning. Activation functions and backpropagation are crucial components that enable these networks to learn complex patterns in data. Understanding these elements is key to grasping how neural networks function and improve over time.

Activation functions introduce non-linearity, allowing networks to model intricate relationships. Backpropagation is the algorithm that powers learning, using gradients to update weights. Together, they form the core of neural network training, enabling these models to tackle diverse machine learning tasks.

Activation functions in neural networks

Types and purposes of activation functions

Top images from around the web for Types and purposes of activation functions

• 

  Common Neural Network Activation Functions - CodeProject View original

  Is this image relevant?

• 

  Activation Functions — The Science of Machine Learning & AI View original

  Is this image relevant?

• 

  Deep Learning (Part 1) - Feedforward neural networks (FNN) View original

  Is this image relevant?

• 

  Common Neural Network Activation Functions - CodeProject View original

  Is this image relevant?

• 

  Activation Functions — The Science of Machine Learning & AI View original

  Is this image relevant?

1 of 3

Top images from around the web for Types and purposes of activation functions

• 

  Common Neural Network Activation Functions - CodeProject View original

  Is this image relevant?

• 

  Activation Functions — The Science of Machine Learning & AI View original

  Is this image relevant?

• 

  Deep Learning (Part 1) - Feedforward neural networks (FNN) View original

  Is this image relevant?

• 

  Common Neural Network Activation Functions - CodeProject View original

  Is this image relevant?

• 

  Activation Functions — The Science of Machine Learning & AI View original

  Is this image relevant?

1 of 3

• Activation functions introduce non-linearity into neural networks, enabling them to learn complex patterns and relationships in data
• The choice of activation function significantly impacts the performance and training dynamics of a neural network
• Activation functions are applied element-wise to the weighted sum of inputs in each neuron, transforming the input signal into an output signal
• Without activation functions, neural networks would be limited to learning linear relationships, severely restricting their representational power

Common activation functions and their properties

• Sigmoid (logistic) activation function squashes the input to a value between 0 and 1, making it suitable for binary classification tasks
• Hyperbolic tangent (tanh) activation function maps the input to a value between -1 and 1, providing a zero-centered output
• Rectified Linear Unit (ReLU) activation function outputs the input if it is positive and zero otherwise, providing faster convergence and alleviating the vanishing gradient problem
• Leaky ReLU addresses the "dying ReLU" problem by allowing small negative values when the input is negative (e.g., 0.01 times the input), maintaining non-zero gradients
• Softmax activation function is commonly used in the output layer for multi-class classification tasks, converting raw scores into a probability distribution over classes

Backpropagation for neural network training

Forward propagation and loss computation

• During forward propagation, the input data is fed through the network, and the activations of each layer are computed using the corresponding weights and activation functions
• The input is multiplied by the weights of the first layer, and the resulting weighted sum is passed through the activation function to obtain the activations of the first hidden layer
• The activations of each subsequent layer are computed similarly, using the activations of the previous layer as input
• The loss function, such as mean squared error or cross-entropy, is evaluated by comparing the network's predictions (output layer activations) with the true labels

Backward propagation and gradient calculation

• In the backward propagation phase, the gradients of the loss function with respect to the weights are computed using the chain rule of calculus
• The chain rule allows the gradients to be decomposed into the product of local gradients at each layer, enabling efficient computation
• The gradients are propagated backward through the network, starting from the output layer and moving towards the input layer
• At each layer, the gradients of the loss with respect to the layer's weights are computed by multiplying the gradients from the previous layer with the local gradients of the current layer
• The gradients are used to update the weights in the direction of steepest descent of the loss function, using an optimization algorithm such as gradient descent

Activation function impact on performance

Vanishing and exploding gradient problems

• Sigmoid and tanh activation functions suffer from the vanishing gradient problem, where gradients become extremely small in deep networks, leading to slow convergence or training stagnation
• The vanishing gradient problem arises because the derivatives of sigmoid and tanh functions are close to zero for large positive or negative inputs, causing the gradients to diminish exponentially as they propagate backward
• Exploding gradients can occur when the weights become very large, causing the gradients to grow exponentially and leading to numerical instability
• ReLU activation mitigates the vanishing gradient problem by providing a constant gradient of 1 for positive inputs, allowing the gradients to flow freely through the network

Sparsity and computational efficiency

• ReLU activation promotes sparsity in the network, as it outputs zero for negative inputs, effectively turning off a portion of the neurons
• Sparsity can lead to more efficient representations and faster computation, as fewer neurons are active during forward and backward propagation
• The linear behavior of ReLU for positive inputs allows for faster convergence compared to sigmoid and tanh functions, which saturate at their extremes
• Leaky ReLU maintains the benefits of ReLU while addressing the "dying ReLU" problem, ensuring that neurons remain active and continue to learn even with negative inputs

Gradient descent for neural network optimization

Variants of gradient descent

• Batch gradient descent computes the gradients and updates the weights using the entire training dataset in each iteration, providing stable convergence but can be computationally expensive for large datasets
• Stochastic gradient descent (SGD) approximates the gradients using a single randomly selected training example in each iteration, providing faster updates but with higher variance and noisier convergence
• Mini-batch gradient descent strikes a balance between batch and stochastic methods by computing gradients over small subsets (mini-batches) of the training data, reducing variance and enabling parallelization
• Mini-batch sizes are typically chosen as powers of 2 (e.g., 32, 64, 128) to optimize memory usage and computational efficiency

Adaptive learning rate methods

• Gradient descent with a fixed learning rate can be sensitive to the choice of the learning rate and may require careful tuning
• Adaptive learning rate methods automatically adjust the learning rate for each weight based on historical gradients, improving convergence speed and stability
• Momentum accumulates past gradients to smooth out oscillations and accelerate convergence in relevant directions
• AdaGrad adapts the learning rate for each weight based on the historical squared gradients, giving larger updates to infrequent features and smaller updates to frequent features
• RMSprop addresses the rapid decay of learning rates in AdaGrad by using a moving average of squared gradients instead of the sum
• Adam (Adaptive Moment Estimation) combines the benefits of momentum and RMSprop, adapting the learning rates based on both the first and second moments of the gradients