This work is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.
CNN Hyperparameter Optimization using Random Grid Coarse-to-fine Search for Face Classification
Corresponding Author(s) : Ade Nurhopipah
Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control,
Vol. 6, No. 1, February 2021
Abstract
Convolutional Neural Network (CNN) is a recently used popular machine learning technique to classify images. However, choosing an optimum and efficient architecture is an inevitable challenge. The research goal was to implement CNN on face classification from low quality CCTV footage. The best model was gained from the hyperparameter optimization process used on CNN structure. The optimized hyperparameters were those connected to the structure network including activation function, the number of kernel, the size of kernel, and the number of nodes on the fully connected layers. Hyperparameter optimization strategy used was random grid coarse-to-fine search optimization approach. This approach combined random search, grid search, and coarse-to-fine technique that was easily and efficiently applied, yet worked well. Exhaustive-random search process was done by evaluating all selected activation functions and choosing another hyperparameters randomly. This was based on the assumption that activation functions were the most related hyperparameter to the model. The SELU activation function used in the next step was the one with the best average performance. Grid coarse-to-fine was conducted to optimize the number of kernel and the number of node on fully connected layer, while grid search was conducted to optimize the kernel size. This process aimed to locate optimal value gradually in hyperparameter which had high-dimensional space. Evaluation of the model resulted from the optimum hyperparameter was 97,56%.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- G. Sreenu and M. A. Saleem Durai, “Intelligent Video Surveillance: a Review through Deep Learning Techniques for Crowd Analysis,” J. Big Data, vol. 6, no. 1, pp. 1–27, 2019. https://doi.org/10.1186/s40537-019-0212-5
- J. Kurniawan, S. G. S. Syahra, C. K. Dewa, and Afiahayati, “Traffic Congestion Detection: Learning from CCTV Monitoring Images using Convolutional Neural Network,” Procedia Comput. Sci., vol. 144, pp. 291–297, 2018. https://doi.org/10.1016/j.procs.2018.10.530
- J. H. Kim, H. G. Hong, and K. R. Park, “Convolutional Neural Network-Based Human Detection in Nighttime Images Using Visible Light Camera Sensors,” Sensors (Switzerland), vol. 17, no. 5, 2017. https://dx.doi.org/10.3390%2Fs17051065
- M. Arsenovic, S. Sladojevic, A. Anderla, and D. Stefanovic, “FaceTime - Deep Learning Based Face Recognition Attendance System,” SISY 2017 - IEEE 15th Int. Symp. Intell. Syst. Informatics, Proc., no. October, pp. 53–57, 2017. https://doi.org/10.1109/SISY.2017.8080587
- D. Acharya, K. Khoshelham, and S. Winter, “Real-time Detection and Tracking of Pedestrians in CCTV Images Using a Deep Convolutional Neural Network,” CEUR Workshop Proc., vol. 1913, no. April, pp. 31–36, 2017.
- H. Choi, “CNN Output Optimization for More Balanced Classification,” Int. J. Fuzzy Log. Intell. Syst., vol. 17, no. 2, pp. 98–106, 2017. http://dx.doi.org/10.5391/IJFIS.2017.17.2.98
- E. Bochinski, T. Senst, and T. Sikora, “Hyper-Parameter Optimization for Convolutional Neural Network Committees Based on Evolutionary Algorithms,” Proc. - Int. Conf. Image Process. ICIP, pp. 3924–3928, 2018. https://doi.org/10.1109/ICIP.2017.8297018
- N. M. Aszemi and P. D. D. Dominic, “Hyperparameter Optimization in Convolutional Neural Network using Genetic Algorithms,” Int. J. Adv. Comput. Sci. Appl., vol. 10, no. 6, pp. 269–278, 2019. https://dx.doi.org/10.14569/IJACSA.2019.0100638
- S. Loussaief and A. Abdelkrim, “Convolutional Neural Network Hyper-Parameters Optimization based on Genetic Algorithms,” Int. J. Adv. Comput. Sci. Appl., vol. 9, no. 10, pp. 252–266, 2018. https://dx.doi.org/10.14569/IJACSA.2018.091031
- J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res., vol. 13, pp. 281–305, 2012.
- J. Wu, X. C. Hao, Z. L. Xiong, and H. Lei, “Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization,” J. Electron. Sci. Technol., vol. 17, no. 1, pp. 26–40, 2019. https://doi.org/10.11989/JEST.1674-862X.80904120
- D. P. Tran, G. N. Nguyen, and V. D. Hoang, “Hyperparameter Optimization for Improving Recognition Efficiency of an Adaptive Learning System,” IEEE Access, vol. 8, no. 1, pp. 160569–160580, 2020. https://doi.org/10.1109/ACCESS.2020.3020930
- J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” Adv. Neural Inf. Process. Syst., vol. 4, pp. 2951–2959, 2012.
- L. Xie and A. Yuille, “Genetic CNN,” Proc. IEEE Int. Conf. Comput. Vis., vol. 2017-Octob, pp. 1388–1397, 2017. https://doi.org/10.1109/ICCV.2017.154
- A. Baldominos, Y. Saez, and P. Isasi, “Hybridizing evolutionary computation and deep neural networks: An approach to handwriting recognition using committees and transfer learning,” Complexity, vol. 2019, 2019. https://doi.org/10.1155/2019/2952304
- D. Motta et al., “Optimization of convolutional neural network hyperparameters for automatic classification of adult mosquitoes,” PLoS One, vol. 15, no. 7, pp. 1–30, 2020. https://doi.org/10.1371/journal.pone.0234959
- R. Andonie and A. C. Florea, “CCC Publications Weighted Random Search for CNN Hyperparameter Optimization,” 2020. https://doi.org/10.15837/ijccc.2020.2.3868
- A. Nurhopipah and A. Harjoko, “Motion Detection and Face Recognition For CCTV Surveillance System,” IJCCS (Indonesian J. Comput. Cybern. Syst., vol. 12, no. 2, p. 107, 2018. https://doi.org/10.22146/ijccs.18198
- A. Nurhopipah and U. Hasanah, “Dataset Splitting Techniques Comparison For Face Classification on CCTV Images,” IJCCS (Indonesian J. Comput. Cybern. Syst., vol. 14, no. 4, pp. 341–352, 2020. https://doi.org/10.22146/ijccs.58092
- Suyanto, Machine Learning Tingkat Dasar dan Lanjut. Bandung: Informatika, 2018.
- Suyanto, K. N. Ramadhan, and S. Mandala, Deep Learning ; Modernisasi Machine Learning untuk Big Data. Bandung: Informatika, 2019.
- Aurélien Géron, “Deep Computer Vision Using Convolutional Neural Network,” in Hands-On Machine Learning with Scikit-Learn, Keras & tensorFlow, 2nd ed., Sebastopol: O’Reilly Media, Inc., 2019, pp. 445–496.
- I. Goodfellow, Y. Bengio, and A. Courville, “Convolutional networks,” in Deep Learning, Cambridge: MIT Press, 2016, pp. 321–359.
- Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. https://doi.org/10.1038/nature14539
- Y. Wang, Y. Li, Y. Song, and X. Rong, “The influence of the activation function in a convolution neural network model of facial expression recognition,” Appl. Sci., vol. 10, no. 5, 2020. https://doi.org/10.3390/app10051897
- R. Hazra, A. Kumar, and B. Baranidharan, “Effect of Various Activation Function on Steering Angle Prediction in CNN based Autonomous Vehicle System,” Int. J. Eng. Adv. Technol., vol. 9, no. 2, pp. 3806–3811, 2019. https://doi.org/10.35940/ijeat.B4017.129219
- S. Wu, G. Wang, P. Tang, F. Chen, and L. Shi, “Convolution with even-sized kernels and symmetric padding,” Adv. Neural Inf. Process. Syst., vol. 32, no. NeurIPS, pp. 1–12, 2019.
- S. H. S. Basha, S. R. Dubey, V. Pulabaigari, and S. Mukherjee, “Impact of fully connected layers on performance of convolutional neural networks for image classification,” Neurocomputing, vol. 378, no. April, pp. 112–119, 2020. https://doi.org/10.1016/j.neucom.2019.10.008
- U. Michelucci, Applied Deep Learning. Dübendorf, Switzerland, 2018. https://doi.org/10.1007/978-1-4842-4976-5
- G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-Normalizing Neural Networks,” in In Advances in Neural Information Processing Systems (NIPS), 2017, pp. 972–981.
References
G. Sreenu and M. A. Saleem Durai, “Intelligent Video Surveillance: a Review through Deep Learning Techniques for Crowd Analysis,” J. Big Data, vol. 6, no. 1, pp. 1–27, 2019. https://doi.org/10.1186/s40537-019-0212-5
J. Kurniawan, S. G. S. Syahra, C. K. Dewa, and Afiahayati, “Traffic Congestion Detection: Learning from CCTV Monitoring Images using Convolutional Neural Network,” Procedia Comput. Sci., vol. 144, pp. 291–297, 2018. https://doi.org/10.1016/j.procs.2018.10.530
J. H. Kim, H. G. Hong, and K. R. Park, “Convolutional Neural Network-Based Human Detection in Nighttime Images Using Visible Light Camera Sensors,” Sensors (Switzerland), vol. 17, no. 5, 2017. https://dx.doi.org/10.3390%2Fs17051065
M. Arsenovic, S. Sladojevic, A. Anderla, and D. Stefanovic, “FaceTime - Deep Learning Based Face Recognition Attendance System,” SISY 2017 - IEEE 15th Int. Symp. Intell. Syst. Informatics, Proc., no. October, pp. 53–57, 2017. https://doi.org/10.1109/SISY.2017.8080587
D. Acharya, K. Khoshelham, and S. Winter, “Real-time Detection and Tracking of Pedestrians in CCTV Images Using a Deep Convolutional Neural Network,” CEUR Workshop Proc., vol. 1913, no. April, pp. 31–36, 2017.
H. Choi, “CNN Output Optimization for More Balanced Classification,” Int. J. Fuzzy Log. Intell. Syst., vol. 17, no. 2, pp. 98–106, 2017. http://dx.doi.org/10.5391/IJFIS.2017.17.2.98
E. Bochinski, T. Senst, and T. Sikora, “Hyper-Parameter Optimization for Convolutional Neural Network Committees Based on Evolutionary Algorithms,” Proc. - Int. Conf. Image Process. ICIP, pp. 3924–3928, 2018. https://doi.org/10.1109/ICIP.2017.8297018
N. M. Aszemi and P. D. D. Dominic, “Hyperparameter Optimization in Convolutional Neural Network using Genetic Algorithms,” Int. J. Adv. Comput. Sci. Appl., vol. 10, no. 6, pp. 269–278, 2019. https://dx.doi.org/10.14569/IJACSA.2019.0100638
S. Loussaief and A. Abdelkrim, “Convolutional Neural Network Hyper-Parameters Optimization based on Genetic Algorithms,” Int. J. Adv. Comput. Sci. Appl., vol. 9, no. 10, pp. 252–266, 2018. https://dx.doi.org/10.14569/IJACSA.2018.091031
J. Bergstra and Y. Bengio, “Random search for hyper-parameter optimization,” J. Mach. Learn. Res., vol. 13, pp. 281–305, 2012.
J. Wu, X. C. Hao, Z. L. Xiong, and H. Lei, “Hyperparameter Optimization for Machine Learning Models Based on Bayesian Optimization,” J. Electron. Sci. Technol., vol. 17, no. 1, pp. 26–40, 2019. https://doi.org/10.11989/JEST.1674-862X.80904120
D. P. Tran, G. N. Nguyen, and V. D. Hoang, “Hyperparameter Optimization for Improving Recognition Efficiency of an Adaptive Learning System,” IEEE Access, vol. 8, no. 1, pp. 160569–160580, 2020. https://doi.org/10.1109/ACCESS.2020.3020930
J. Snoek, H. Larochelle, and R. P. Adams, “Practical Bayesian optimization of machine learning algorithms,” Adv. Neural Inf. Process. Syst., vol. 4, pp. 2951–2959, 2012.
L. Xie and A. Yuille, “Genetic CNN,” Proc. IEEE Int. Conf. Comput. Vis., vol. 2017-Octob, pp. 1388–1397, 2017. https://doi.org/10.1109/ICCV.2017.154
A. Baldominos, Y. Saez, and P. Isasi, “Hybridizing evolutionary computation and deep neural networks: An approach to handwriting recognition using committees and transfer learning,” Complexity, vol. 2019, 2019. https://doi.org/10.1155/2019/2952304
D. Motta et al., “Optimization of convolutional neural network hyperparameters for automatic classification of adult mosquitoes,” PLoS One, vol. 15, no. 7, pp. 1–30, 2020. https://doi.org/10.1371/journal.pone.0234959
R. Andonie and A. C. Florea, “CCC Publications Weighted Random Search for CNN Hyperparameter Optimization,” 2020. https://doi.org/10.15837/ijccc.2020.2.3868
A. Nurhopipah and A. Harjoko, “Motion Detection and Face Recognition For CCTV Surveillance System,” IJCCS (Indonesian J. Comput. Cybern. Syst., vol. 12, no. 2, p. 107, 2018. https://doi.org/10.22146/ijccs.18198
A. Nurhopipah and U. Hasanah, “Dataset Splitting Techniques Comparison For Face Classification on CCTV Images,” IJCCS (Indonesian J. Comput. Cybern. Syst., vol. 14, no. 4, pp. 341–352, 2020. https://doi.org/10.22146/ijccs.58092
Suyanto, Machine Learning Tingkat Dasar dan Lanjut. Bandung: Informatika, 2018.
Suyanto, K. N. Ramadhan, and S. Mandala, Deep Learning ; Modernisasi Machine Learning untuk Big Data. Bandung: Informatika, 2019.
Aurélien Géron, “Deep Computer Vision Using Convolutional Neural Network,” in Hands-On Machine Learning with Scikit-Learn, Keras & tensorFlow, 2nd ed., Sebastopol: O’Reilly Media, Inc., 2019, pp. 445–496.
I. Goodfellow, Y. Bengio, and A. Courville, “Convolutional networks,” in Deep Learning, Cambridge: MIT Press, 2016, pp. 321–359.
Y. Lecun, Y. Bengio, and G. Hinton, “Deep learning,” Nature, vol. 521, no. 7553, pp. 436–444, 2015. https://doi.org/10.1038/nature14539
Y. Wang, Y. Li, Y. Song, and X. Rong, “The influence of the activation function in a convolution neural network model of facial expression recognition,” Appl. Sci., vol. 10, no. 5, 2020. https://doi.org/10.3390/app10051897
R. Hazra, A. Kumar, and B. Baranidharan, “Effect of Various Activation Function on Steering Angle Prediction in CNN based Autonomous Vehicle System,” Int. J. Eng. Adv. Technol., vol. 9, no. 2, pp. 3806–3811, 2019. https://doi.org/10.35940/ijeat.B4017.129219
S. Wu, G. Wang, P. Tang, F. Chen, and L. Shi, “Convolution with even-sized kernels and symmetric padding,” Adv. Neural Inf. Process. Syst., vol. 32, no. NeurIPS, pp. 1–12, 2019.
S. H. S. Basha, S. R. Dubey, V. Pulabaigari, and S. Mukherjee, “Impact of fully connected layers on performance of convolutional neural networks for image classification,” Neurocomputing, vol. 378, no. April, pp. 112–119, 2020. https://doi.org/10.1016/j.neucom.2019.10.008
U. Michelucci, Applied Deep Learning. Dübendorf, Switzerland, 2018. https://doi.org/10.1007/978-1-4842-4976-5
G. Klambauer, T. Unterthiner, A. Mayr, and S. Hochreiter, “Self-Normalizing Neural Networks,” in In Advances in Neural Information Processing Systems (NIPS), 2017, pp. 972–981.