Article Sidebar
Main Article Content
The performance of convolutional neural networks (CNN) for computer vision problems depends heavily on their architectures. Transfer learning performance of a CNN strongly relies on selection of its trainable layers. Selecting the most effective update layers for a certain target dataset often requires expert knowledge on CNN architecture which many practitioners do not possess. General users prefer to use an available architecture (e.g. GoogleNet, ResNet, EfficientNet etc.) that is developed by domain experts. With the ever-growing number of layers, it is increasingly becoming difficult and cumbersome to handpick the update layers. Therefore, in this paper we explore the application of a genetic algorithm to mitigate this problem. The convolutional layers of popular pre-trained networks are often grouped into modules that constitute their building blocks. We devise a genetic algorithm to select blocks of layers for updating the parameters. By experimenting with EfficientNetB0 pre-trained on ImageNet and using three popular image datasets - namely Food-101, CIFAR-100 and MangoLeafBD - as target datasets, we show that our algorithm yields similar or better results than the baseline in terms of accuracy, and requires lower training and evaluation time due to learning a smaller number of parameters. We also devise a measure called block importance to measure each block's efficacy as an update block and analyze the importance of the blocks selected by our algorithm.
Article Details
S. I. Ahmed, M. Ibrahim, M. Nadim, M. M. Rahman, M. M. Shejunti, et al. MangoLeafBD: A comprehensive image dataset to classify diseased and healthy mango leaves. Data in Brief, 47:108941, 2023. https://doi.org/10.1016/j.dib.2023.108941. (Crossref)
D. Alvarez-Melis and N. Fusi. Geometric dataset distances via optimal transport. In: Proc. 33th Int. Conf. Neural Information Processing Systems (NeurIPS), pp. 21428-21439. Curran Associates, 2020. https://proceedings.neurips.cc/paper_files/paper/2020/hash/f52a7b2610fb4d3f74b4106fb80b233d-Abstract.html.
D. Alvarez-Melis, Microsoft Open Source, and C. Yang. microsoft / otdd. GitHub. https://github.com/microsoft/otdd.
N. M. Aszemi and P. D. D. Dominic. Hyperparameter optimization in convolutional neural network using genetic algorithms. International Journal of Advanced Computer Science and Applications, 10(6):269-278, 2019. https://doi.org/10.14569/IJACSA.2019.0100638. (Crossref)
E. Bochinski, T. Senst, and T. Sikora. Hyper-parameter optimization for convolutional neural network committees based on evolutionary algorithms. In: Proc. 2017 IEEE International Conference on Image Processing (ICIP), pp. 3924-3928, 2017. https://doi.org/10.1109/ICIP.2017.8297018. (Crossref)
L. Bossard, M. Guillaumin, and L. V. Gool. Food-101 - Mining discriminative components with random forests. In: Proc. European Conference on Computer Vision (ECCV 2014), pp. 446-461, 2014. https://doi.org/https://doi.org/10.1007/978-3-319-10599-4_29. (Crossref)
R. Cai and J. Luo. Multi-task learning for multi-objective evolutionary neural architecture search. In: Proc. 2021 IEEE Congress on Evolutionary Computation (CEC), pp. 1680-1687, 2021. https://doi.org/10.1109/CEC45853.2021.9504721. (Crossref)
D. J. Dong, W. Socher, R. Li, Li-Jia, K. Li, et al. ImageNet: A large-scale hierarchical image database. In: Proc. 2009 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 248-255, 2009. https://doi.org/10.1109/CVPR.2009.5206848. (Crossref)
C. Fernando, D. Banarse, C. Blundell, Y. Zwols, D. Ha, et al. Pathnet: Evolution channels gradient descent in super neural networks. arXiv, 2017. ArXiv.1701.08734. https://doi.org/10.48550/ARXIV.1701.08734.
M. Ghafoorian, A. Mehertash, T. Kapur, N. Karssemeijer, E. Marchiori, et al. Transfer learning for domain adaptation in mri: Application in brain lesion segmentation. In: Proc. International Conference on Medical Image Computing and Computer-Assisted Intervention (MICCAI), p. 516–524, 2017. https://doi.org/https://doi.org/10.1007/978-3-319-66179-7_59. (Crossref)
D. E. Goldberg. Genetic Algorithms. Pearson Education India, 2013.
M. S. Haque, M. Fahim, and M. Ibrahim. An exploratory study on simulated annealing for feature selection in learning-to-rank. International Journal of Intelligent Systems and Applications (IJISA), 16:86-103, 2024. https://doi.org/10.5815/ijisa.2024.04.06. (Crossref)
K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 770–778, 2016. https://doi.org/10.1109/CVPR.2016.90. (Crossref)
G. Huang, Z. Liu, K. Q. Weinberger, and L. van der Maaten. Densely connected convolutional networks. Proc. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), p. 2261–2269, 2017. https://doi.org/10.1109/CVPR.2017.243. (Crossref)
Y. Huang, Y. Cheng, D. Chen, H. Lee, J. Ngiam, et al. GPipe: Efficient training of giant neural networks using pipeline parallelism. arXiv, 2019. ArXiv:1811.06965v5. https://doi.org/10.48550/arXiv.1811.06965.
S. Imai, S. Kawai, and H. Nobuhara. Stepwise pathnet: A layer-by-layer knowledge selection-based transfer learning algorithm. Scientific Reports, 10:8132, 2020. https://doi.org/10.1038/s41598-020-64165-3. (Crossref)
A. Krizhevsky. Learning multiple layers of features from tiny images. M.sc. thesis, University of Toronto, 2009. https://www.cs.toronto.edu/~kriz/learning-features-2009-TR.pdf.
A. Krizhevsky, I. Sutskever, and G. E. Hinton. ImageNet classification with deep convolutional neural networks. In: Proc. Advances in Neural Information Processing Systems 25 (NIPS), p. 1097–1105. Curran Associates, 2012. https://proceedings.neurips.cc/paper/2012/hash/c399862d3b9d6b76c8436e924a68c45b-Abstract.html.
S. Lee, J. Kim, H. Kang, D.-Y. Kang, and J. Park. Genetic algorithm based deep learning neural network structure and hyperparameter optimization. Applied Sciences, 11(2):744, 2021. https://doi.org/10.3390/app11020744. (Crossref)
S. Loussaief and A. Abdelkrim. Convolutional neural network hyper-parameters optimization based on genetic algorithms. International Journal of Advanced Computer Science and Applications, 9(10), 2018. https://doi.org/10.14569/IJACSA.2018.091031. (Crossref)
Z. Lu, H. Pu, F. Wang, Z. Hu, and L. Wang. The expressive power of neural networks: A view from the width. In: Proc. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. https://proceedings.neurips.cc/paper/2017/hash/32cbf687880eb1674a07bf717761dd3a-Abstract.html.
Z. Michalewicz. Genetic Algorithms + Data Structures = Evolution Programs. Springer, Berlin, Heidelberg, 2013. https://doi.org/10.1007/978-3-662-07418-3. (Crossref)
A. Mimi, S. F. T. Zohura, M. Ibrahim, R. R. Haque, O. Farrok, et al. Identifying selected diseases of leaves using deep learning and transfer learning models. Machine Graphics and Vision, 32(1):55-71, 2023. https://doi.org/10.22630/MGV.2023.32.1.3. (Crossref)
S. Nagae, D. Kanda, S. Kawa, and H. Nobuhara. Automatic layer selection for transfer learning and quantitative evaluation of layer effectiveness. Neurocomputing, 469:151-162, 2022. https://doi.org/10.1016/j.neucom.2021.10.051. (Crossref)
S. J. Pan and Q. Yang. A survey on transfer learning. IEEE Transactions on Knowledge and Data Engineering, 22(10):1345-1359, 2010. https://doi.org/10.1109/TKDE.2009.191. (Crossref)
M. Raghu, B. Poole, J. Kleinberg, S. Ganguli, and J. SohlDickstein. On the expressive power of deep neural networks. In: Proc. Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017. https://proceedings.neurips.cc/paper/2017/hash/32cbf687880eb1674a07bf717761dd3a-Abstract.html.
H. Rehana, M. Ibrahim, and M. H. Ali. Plant disease detection using region-based convolutional neural network. arXiv, 2023. ArXiv.10.48550. https://doi.org/10.48550/arXiv.2303.09063.
K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In: Proc. 3rd International Conference on Learning Representations (ICLR), 2015. Accessible in arXiv. https://doi.org/10.48550/arXiv.1409.1556.
Y. Sun, B. Xue, M. Zhang, and G. G. Yen. Automatically evolving CNN architectures based on blocks. arXiv, 2019. ArXiv.1810.11875v2. https://doi.org/10.48550/arXiv.1810.11875v2.
C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed, et al. Going deeper with convolutions. In: Proc. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2015. https://doi.org/10.1109/CVPR.2015.7298594. (Crossref)
C. Szegedy, V. Vanhoucke, S. Ioffe, J. Shlens, and Z. Wojna. Rethinking the inception architecture for computer vision. In: Proc. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2818-2826, 2016. https://doi.org/10.1109/CVPR.2016.308. (Crossref)
M. Tan and Q. V. Le. EfficientNet: Rethinking model scaling for convolutional neural networks. In: Proc. International Conference on Machine Learning (ICML), vol. 97 of Proceedings of Machine Learning Research, p. 6105–6114, 2019. https://proceedings.mlr.press/v97/tan19a.html.
H. Tian, S. Pouyanfar, J. Chen, S. Chen, and S. S. Iyengar. Automatic convolutional neural network selection for image classification using genetic algorithms. Proc. 2018 IEEE International Conference on Information Reuse and Integration (IRI), pp. 444-451, 2018. https://doi.org/10.1109/IRI.2018.00071. (Crossref)
C. Villani. Optimal transport, Old and New. Springer, 2008. https://doi.org/10.1007/978-3-540-71050-9. (Crossref)
L. Xie and A. Yuille. Genetic CNN. In: Proc. 2017 IEEE International Conference on Computer Vision (ICCV), pp. 1388-1397, 2017. https://doi.org/10.1109/ICCV.2017.154. (Crossref)
S. Zagoruyko and N. Komodakis. Wide residual networks. In: Proc. British Machine Vision Conference (BMVC), pp. 87.1-87.12, 2016. https://bmva-archive.org.uk/bmvc/2016/papers/paper087/index.html. (Crossref)
M. D. Zeiler and R. Fergus. Visualizing and understanding convolutional networks. In: Proc. European Conference on Computer Vision (ECCV 2014), vol. 8689 of Lecture Notes in Computer Science, p. 818–833, 2014. https://doi.org/10.1007/978-3-319-10590-1_53. (Crossref)
H. Zunair, N. Mohammed, and S. Momen. Unconventional wisdom: A new transfer learning approach applied to Bengali numeral classification. Proc. 2018 International Conference on Bangla Speech and Language Processing (ICBSLP), 21:1-6, 2018. https://doi.org/10.1109/ICBSLP.2018.8554435. (Crossref)
Downloads
- Afsana Mimi, Sayeda Fatema Tuj Zohura, Muhammad Ibrahim, Riddho Ridwanul Haque, Omar Farrok, Taskeed Jabid, Md Sawkat Ali, Identifying selected diseases of leaves using deep learning and transfer learning models , Machine Graphics and Vision: Vol. 32 No. 1 (2023)