7.3 Popular CNN architectures: AlexNet, VGG, ResNet, and Inception

Convolutional Neural Networks (CNNs) have revolutionized computer vision. Key architectures like AlexNet, VGG, ResNet, and Inception have pushed the boundaries of image recognition, introducing innovations that shaped the field.

These architectures brought game-changing ideas: ReLU activations, dropout regularization, deep networks with small filters, residual connections, and parallel convolution paths. Each design tackled specific challenges, paving the way for more powerful and efficient image processing systems.

Popular CNN Architectures

Key innovations of AlexNet

Top images from around the web for Key innovations of AlexNet

• 

  [DL] CNN - AlexNet (2012) View original

  Is this image relevant?

• 

  AlexNet | 深層学習の概念および畳み込みニューラルネットワークの概念を初めて取り入れたアーキテクチャ View original

  Is this image relevant?

• 

  深度学习之CNN模型演化 | 杨青海的博客 View original

  Is this image relevant?

• 

  [DL] CNN - AlexNet (2012) View original

  Is this image relevant?

• 

  AlexNet | 深層学習の概念および畳み込みニューラルネットワークの概念を初めて取り入れたアーキテクチャ View original

  Is this image relevant?

1 of 3

Top images from around the web for Key innovations of AlexNet

• 

  [DL] CNN - AlexNet (2012) View original

  Is this image relevant?

• 

  AlexNet | 深層学習の概念および畳み込みニューラルネットワークの概念を初めて取り入れたアーキテクチャ View original

  Is this image relevant?

• 

  深度学习之CNN模型演化 | 杨青海的博客 View original

  Is this image relevant?

• 

  [DL] CNN - AlexNet (2012) View original

  Is this image relevant?

• 

  AlexNet | 深層学習の概念および畳み込みニューラルネットワークの概念を初めて取り入れたアーキテクチャ View original

  Is this image relevant?

1 of 3

• ReLU activation revolutionized neural networks with non-linear $f(x) = max(0, x)$ function speeding up training and reducing vanishing gradient problem
• Dropout regularization randomly deactivates neurons during training forcing network to learn robust features and improving generalization
• GPU implementation utilized NVIDIA GTX 580 GPUs enabled training larger models on ImageNet dataset
• Local Response Normalization applied after ReLU in certain layers aided generalization and reduced overfitting

Design principles of VGG

• Small 3x3 convolutional filters throughout network increased non-linearity and reduced parameters
• Deep architecture with VGG-16 and VGG-19 variants demonstrated power of network depth in improving performance
• Consistent pattern of repeating convolutional blocks followed by max pooling simplified network design and analysis
• Fully connected layers at end with 4096 channels in first two and 1000 in last for ImageNet classification
• Pre-training and fine-tuning introduced training shallower versions first using weights to initialize deeper ones

Residual connections in ResNet

• Skip connections allow information to bypass layers with formula $y = F(x) + x$ where $F(x)$ is residual function
• Addressed degradation problem in very deep networks enabling training of 100+ layer architectures
• Easier optimization as residual connections help network learn identity mappings
• Improved gradient flow mitigated vanishing gradient problem in deep networks
• Bottleneck architecture used 1x1 convolutions to reduce and increase dimensions reducing computational complexity

Inception architecture and parallel paths

• Multiple convolution operations performed in parallel with outputs concatenated
• Various filter sizes (1x1, 3x3, 5x5) captured features at different scales simultaneously
• 1x1 convolutions reduced dimensionality and computational cost before larger convolutions
• Pooling path included parallel max pooling operation retained important features while reducing spatial dimensions
• Inception modules as building blocks stacked to form complete architecture
• Global Average Pooling replaced fully connected layers at end reducing parameters and preventing overfitting