1.2 Key concepts and terminology in deep learning

Neural networks are the backbone of deep learning, mimicking the human brain's structure. They consist of interconnected neurons organized in layers, using activation functions to process information and learn complex patterns from data.

Deep learning approaches include supervised, unsupervised, and reinforcement learning, each suited for different tasks. The workflow involves carefully splitting datasets for training, validation, and testing, while gradient descent and backpropagation drive the learning process.

Fundamental Concepts in Deep Learning

Fundamentals of neural networks

Top images from around the web for Fundamentals of neural networks

• 

  Common Neural Network Activation Functions - CodeProject View original

  Is this image relevant?

• 

  Introduction to Artificial Neural Networks - CodeProject View original

  Is this image relevant?

• 

  Understanding Neural Networks: What, How and Why? – Towards Data Science View original

  Is this image relevant?

• 

  Common Neural Network Activation Functions - CodeProject View original

  Is this image relevant?

• 

  Introduction to Artificial Neural Networks - CodeProject View original

  Is this image relevant?

1 of 3

Top images from around the web for Fundamentals of neural networks

• 

  Common Neural Network Activation Functions - CodeProject View original

  Is this image relevant?

• 

  Introduction to Artificial Neural Networks - CodeProject View original

  Is this image relevant?

• 

  Understanding Neural Networks: What, How and Why? – Towards Data Science View original

  Is this image relevant?

• 

  Common Neural Network Activation Functions - CodeProject View original

  Is this image relevant?

• 

  Introduction to Artificial Neural Networks - CodeProject View original

  Is this image relevant?

1 of 3

• Artificial neurons mimic biological neurons receive and process inputs produce outputs form building blocks of neural networks
  • Dendrites (input connections)
  • Cell body (processes inputs)
  • Axon (output connection)
• Activation functions introduce non-linearity enable networks to learn complex patterns transform neuron outputs
  • Sigmoid squashes values between 0 and 1 (logistic function)
  • ReLU outputs input if positive otherwise zero addresses vanishing gradient problem
  • Tanh maps inputs to values between -1 and 1 zero-centered
• Layers organize neurons into functional groups process information hierarchically
  • Input layer receives raw data (image pixels, text tokens)
  • Hidden layers extract features learn representations (edge detectors, semantic concepts)
  • Output layer produces final predictions (class probabilities, regression values)
• Network architectures determine overall structure and information flow
  • Feedforward networks process data in one direction (image classification)
  • CNNs use convolutional layers for spatial data (object detection, image segmentation)
  • RNNs handle sequential data with feedback loops (language translation, time series forecasting)

Types of deep learning approaches

• Supervised learning uses labeled data pairs input with corresponding output
  • Classification assigns inputs to predefined categories (spam detection, sentiment analysis)
  • Regression predicts continuous values (house price prediction, stock market forecasting)
• Unsupervised learning discovers patterns in unlabeled data
  • Clustering groups similar data points (customer segmentation, anomaly detection)
  • Dimensionality reduction compresses high-dimensional data (feature extraction, data visualization)
• Reinforcement learning agent interacts with environment learns optimal behavior through trial and error
  • Markov Decision Process models decision-making
  • Q-learning estimates action values
  • Policy gradients directly optimize decision-making policy

Datasets in deep learning workflow

• Training dataset largest portion (~60-80%) used to update model parameters
  • Gradient descent minimizes loss function
  • Backpropagation computes gradients
• Validation dataset separate subset (~10-20%) used for hyperparameter tuning and model selection
  • Learning rate adjustment
  • Architecture modifications
  • Early stopping to prevent overfitting
• Testing dataset completely unseen data (~10-20%) evaluates final model performance
  • Generalization assessment
  • Unbiased performance metrics (accuracy, F1-score, mean squared error)

Gradient descent for network training

• Gradient descent iteratively updates model parameters to minimize loss function
  • Computes gradient of loss with respect to parameters
  • Updates parameters in opposite direction of gradient
• Importance in training enables networks to learn complex patterns from data
  • Automatic feature extraction
  • End-to-end learning without manual feature engineering
• Gradient descent variants balance computational efficiency and convergence
  • Batch processes entire dataset per update (stable but slow)
  • Stochastic (SGD) uses single sample per update (noisy but fast)
  • Mini-batch compromise between batch and SGD (most common approach)
• Learning rate hyperparameter controls step size during optimization
  • Too high leads to divergence
  • Too low results in slow convergence
  • Adaptive methods (Adam, RMSprop) automatically adjust learning rates
• Backpropagation efficiently computes gradients through chain rule
  • Forward pass calculates activations
  • Backward pass propagates error gradients
• Optimization challenges in deep networks
  • Vanishing gradients impede learning in early layers
  • Exploding gradients cause instability
  • Local minima and saddle points slow convergence