AI seminar : light weight network[1] – CNN (2/2) -> intuitive explanation

Exploring advancements in lightweight CNNs: diving into the roles of Dense Convolution, Shift Convolution, inverted residual block, Squeeze-and-Excitation

5. Dense Convolution

Huang, G., Liu, Z., Van Der Maaten, L. and Weinberger, K.Q., 2017. Densely connected convolutional networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4700-4708). cited 38142

• Traditional Challenges:
  • Past research ([1.]) combined data additively, which might delay information flow.
• The Concatenation Method:
  • This study introduces concatenation as a way to merge previous information.
  • Benefits include clear distinction and organization of data.
• Preventing Data Loss:
  • Using concatenation safeguards against potential information loss.
• Performance Improvements:
  • Models utilizing this approach outperform traditional ones.
  • They achieve this with fewer parameters, indicating efficiency.
• Mechanics of Implementation:
  • The technique concatenates a specific layer’s input with subsequent layer inputs.
• Personal Note:
  • I have yet to experiment with this method personally, but its potential is evident.

6. Shift Convolution

Wu, Bichen, et al. “Shift: A zero flop, zero parameter alternative to spatial convolutions.” Proceedings of the IEEE conference on computer vision and pattern recognition. 2018. cited 367

• Revisiting Previous Methods:
  • Replaces the traditional convolution method, specifically the “3. Depthwise Conv”, with shift convolution operations.
• Advantages of Shift Convolution:
  • Enhances computational efficiency.
  • Reduces the number of parameters.
• Personal Implementation Experience:
  • I have yet to experiment with this technique personally.
• Application in Our Research:
  1. After applying “3. Depthwise Conv”, assess performance improvements.
  2. Test by individually applying “3. Depthwise Conv” and “4. (not specified in the text)” to gauge performance enhancements.
  3. Evaluate performance improvements when “3. Depthwise Conv” is substituted with “6. Shift Convolution”.
  4. If no improvement is observed after applying “3. Depthwise Conv”, then:
     • Apply “5. (not specified in the text)” to check for performance enhancements.
     • Subsequently, apply “4. (not specified in the text)” to further assess the performance gains.

7. Inverted residual block

Sandler, M., Howard, A., Zhu, M., Zhmoginov, A. and Chen, L.C., 2018. Mobilenetv2: Inverted residuals and linear bottlenecks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 4510-4520). cited 17418

• Relation to Previous Methods:
  • An advanced methodology developed subsequent to “1. (not specified)”.
• Origin & Purpose:
  • Introduced by Google in 2018 as part of a lightweight deep learning architecture.
  • Primary objective: Minimize computational demands while maintaining high accuracy.
• Distinguishing Structural Feature:
  • It adopts a (narrow -> wide -> narrow) structure for input.
  • This is a contrast to “1. (not specified)” which uses a (wide -> narrow -> wide) structure.
• Advantages:
  • Enhanced memory efficiency.
  • Reduction in the number of parameters.
  • Performance improvement.
• Personal Note on Implementation:
  • I haven’t had the experience of personally implementing this technique.

8. Squeeze-and-Excitation

Hu, J., Shen, L. and Sun, G., 2018. Squeeze-and-excitation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 7132-7141). cited 24119

• Positioning Against Previous Research:
  • Not necessarily a direct solution to previous research limitations but an approach for performance enhancement.
• Core Principle:
  • Trains the network to discern which channels (or what kind of information) are pivotal.
  • Achieves this by multiplying weights and learning optimal values for those weights.
• Implementation Experience:
  • Yes, I’ve had the opportunity to experiment with this technique.
• Relevance to Our Research:
  1. Semantics & Weights: The technique inherently determines correlations in input data and identifies significant patterns. By assigning weights based on these patterns, there’s potential for performance improvement.
  2. Practical Application: Without a detailed review, we directly applied it to all convolution layers in our CNN (dilated). This immediate application did not yield any notable performance boost.

Conclusion

Introduction: Setting the benchmark with CNNs has revolutionized our understanding of machine capabilities, often surpassing human-level performance. The next logical phase is transferring this heightened performance to computing-constrained environments, such as embedded devices.

Our Research Direction:

• Why Lightweight Networks?
  With embedded systems, like the Jetson Nano, having limited computational power, there’s an imperative to maintain performance while being resource-friendly.
• Layers in Focus:
  Our research primarily concentrates on architectures with around 6 layers.

Insights from Past Research:

• Enhanced Layer Structures:
  Previous works have introduced certain layer formats that promise increased performance. Implementing and testing these could provide pivotal insights for our studies.
• Information Flow Patterns:
  Besides structure, how information travels and is processed in the network is equally consequential. We aim to draw from such patterns to understand potential optimizations.
• Resource Optimization:
  The focus isn’t just on performance but on doing more with less. Research to reduce computational demands and the number of parameters can be especially beneficial for devices with restricted computing power.

Our Preliminary Findings:

• Application on Current Model:
  By adapting these lightweight strategies to our existing CNN model (dilated, 6 layers), we anticipate maintaining or even enhancing its performance.

The Future: A Trend to Watch

• Reinforcement Learning in Lightweight Networks:
  As we move forward, there’s a rising trend towards networks that utilize reinforcement learning. This self-optimizing nature ensures that the network identifies its optimal structure, making it a promising avenue for future exploration.