This work is licensed under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.
Neural Network-Based Image Processing for Tomato Harvesting Robot
Kinetik: Game Technology, Information System, Computer Network, Computing, Electronics, and Control,
Vol. 8, No. 3, August 2023
Abstract
Agriculture is one of the areas that can benefit from robotics technology, as it faces issues such as a shortage of human labor and access to less arid terrain. Harvesting is an important step in agriculture since workers are required to work around the clock. The red ripe tomatoes should go to the nearest market, while the greenest should go to the farthest market. Harvesting robots can benefit from Neural Network-based image processing to ensure robust detection. The vision system should assist the mobility system in moving precisely and at the appropriate speed. The design and implementation of a harvesting robot are described in this study. The efficiency of the proposed strategy is tested by picking red-ripened tomatoes while leaving the yellowish ones out of the experimental test bed. The experiment results demonstrate that the effectiveness of the proposed method in harvesting the right tomatoes is 80%.
Keywords
Download Citation
Endnote/Zotero/Mendeley (RIS)BibTeX
- M. Stoelen et al., "Low-Cost Robotics for Horticulture: A Case Study on Automated Sugar Pea Harvesting," 10th European Conference on Precision Agriculture (ECPA), 2015. https://doi.org/10.3920/978-90-8686-814-8
- T. Dewi, S. Nurmaini, P., Risma, and Y. Oktarina, Y., “Inverse Kinematic Analysis of 4 DOF Pick and Place Arm Robot Manipulator using Fuzzy Logic Controller,” International Journal of Electrical and Computer Engineering (IJECE), Vol. 10, No 2, pp. 1376-1386, 2020. http://doi.org/10.11591/ijece.v10i2.pp1376-1386
- T. Dewi, C. Anggraini, P. Risma, Y. Oktarina, and Muslikhin, “Motion Control Analysis of Two Collaborative Arm Robots in Fruit Packaging System,” SINERGIA Vol. 25, No. 2, pp. 217-226, 2021. http://doi.org/10.22441/sinergi.2021.2.013
- T. Dewi, Z. Mulya, P. Risma, and Y. Oktarina, “BLOB Analysis of an Automatic Vision Guided System for a Fruit Picking and Placing Robot,” International Journal of Computational Vision and Robotics, Vol. 11, No 3, pp. 315-326, 2021. https://doi.org/10.1504/IJCVR.2021.115161
- C. Wang, Y. Tang, X. Zou, W. SiTu, and W. Feng, "A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot," Optik, Vol. 131, pp. 626-631, 2017. https://doi.org/10.1016/j.ijleo.2016.11.177.
- C.W. Bac, J. Hemming, and E.J. Van Henten, “Robust pixel-based classification of obstacles for robotic harvesting of sweet-pepper,” Computers and Electronics in Agriculture, Vol. 96, pp. 148-162, 2013. https://doi.org/10.1016/j.compag.2013.05.004
- N. M. Syahrian, P. Risma, and T. Dewi, “Vision-Based Pipe Monitoring Robot for Crack Detection using Canny Edge Detection Method as an Image Processing Technique,” Kinetik: Game Technology, Information System, Computer Network, Computing Electronics, and Control, Vol. 2, No. 4, pp. 243-250, 2017. https://doi.org/10.22219/kinetik.v2i4.243
- M.D. Yusuf, RD. Kusumanto, Y. Oktarina, T. Dewi, and P. Risma, “Blob Analysis for Fruit Recognition and Detection,” Computer Engineering and Applications, Vol 7 No 1 pp. 23-32, 2018. https://doi.org/10.18495/comengapp.v7i1.237
- T. Dewi, P. Risma, Y. Oktarina and S. Muslimin, "Visual Servoing Design and Control for Agriculture Robot; a Review," 2018 International Conference on Electrical Engineering and Computer Science (ICECOS), Pangkal, Indonesia, 2018, pp. 57-62. https://doi.org/10.1109/ICECOS.2018.8605209
- H. Gharakhani, J. A. Thomasson, and Y. Lu, "An end-effector for robotic cotton harvesting," Smart Agricultural Technology, Vol. 2, p. 100043, 2022. https://doi.org/10.1016/j.atech.2022.100043.
- T. Dewi, P. Risma, Y. Oktarina, and M. Nawawi, “Tomato Harvesting Arm Robot Manipulator; a Pilot Project,” Journal of Physics: Conference Series, 1500, p 012003, Proc. 3rd FIRST, Palembang: Indonesia, 2020. https://doi.org/10.1088/1742-6596/1500/1/012003
- Z. Hou, Z. Li, T. Fadiji, and J. Fu,Soft, “Grasping Mechanism of Human Fingers for Tomato-picking Bionic Robots,” Computers and Electronics in Agriculture, Vol 182, 106010, 2021. https://doi.org/10.1016/j.compag.2021.106010
- J. Chen, H. Qiang, J. Wu, G. Xu, and Z. Wang, “Navigation Path Extraction for Greenhouse Cucumber-picking Robots Using the Prediction-point Hough Transform, Computers and Electronics in Agriculture,” Vol. 180, 105911, 2021. https://doi.org/10.1016/j.compag.2020.105911
- L. van Herck, P. Kurtser, L. Wittemans, and Y. Edan, “Crop Design for Improved Robotic Harvesting: a Case Study of Sweet Pepper Harvesting, Biosystems Engineering,” Vol 192, pp. 294-308, 2020. https://doi.org/10.1016/j.biosystemseng.2020.01.021
- Y. Zhao, L. Gong, C. Liu, and Y. Huang, "Dual-arm Robot Design and Testing for Harvesting Tomato in Greenhouse," IFAC-PapersOnLine, Vol. 49, No 16, pp. 161-165, 2016. https://doi.org/10.1016/j.ifacol.2016.10.030
- T. Dewi, P. Risma, and Y. Oktarina, “Fruit Sorting Robot based on Color and Size for an Agricultural Product Packaging System,” Bulletin of Electrical Engineering, and Informatics (BEEI), vol. 9, no. 4, pp. 1438-1445, 2020. https://doi.org/10.11591/eei.v9i4.2353
- A. Nasiri, A. Taheri-Garavand, and Y. Zhang Image-based deep learning automated sorting of date fruit, Postharvest Biology and Technology, Vol. 153, pp. 133-141, 2019. https://doi.org/10.1016/j.postharvbio.2019.04.003.
- M. Fashi, L. Naderloo, and H. Javadikia, The relationship between the appearance of pomegranate fruit and color and size of arils based on image processing, Postharvest Biology and Technology, Vol. 154, pp. 52-57, 2019. https://doi.org/10.1016/j.postharvbio.2019.04.017.
- J. Jhawar, “Orange Sorting by Applying Pattern Recognition on Colour Image,” Procedia Computer Science, vol. 78, pp. 691–697, December 2016. https://doi.org/10.1016/j.procs.2016.02.118
- L. Fu, J. Duan, X. Zou, G. Lin, S. Song, B. Ji, and Z. Yang, Banana detection based on color and texture features in the natural environment, Computers and Electronics in Agriculture, Vol. 167, p. 105057, 2019. https://doi.org/10.1016/j.compag.2019.105057.
- U. Dorj, M. Lee, and S. Yun, An yield estimation in citrus orchards via fruit detection and counting using image processing, Computers and Electronics in Agriculture, Vol. 140, pp. 103-112, 2017. https://doi.org/10.1016/j.compag.2017.05.019.
- L. Fu, Z. Liu, Y. Majeed, and Y. Cui, “Kiwifruit Yield Estimation using Processing by an Android Mobile Phone,” IFAC Conference Paper Archive, vol. 51, pp. 185–190, 2018. https://doi.org/10.1016/j.ifacol.2018.08.137
- K. Tan, W. Suk, H. Gan, and S. Wang, “Recognising Blueberry Fruit of Different Maturity Using Histogram Oriented Gradients and Colour Features in Outdoor Scenes,” Biosystems Engineering, vol. 176, pp. 59–72, 2018. https://doi.org/10.1016/j.biosystemseng.2018.08.011
- A. Septiarini, H. Hamdani, H. R. Hatta, and K. Anwar, "Automatic Image Segmentation of Oil Palm Fruits by Applying the Contour-Based Approach," Scientia Horticulturae, 2019. https://doi.org/10.1016/j.scienta.2019.108939.
- M. H. Malik, T. Zhang, H. Li, M. Zhang, S. Shabbir, and A. Saeed, "Mature Tomato Fruit Detection Algorithm Based on improved HSV and Watershed Algorithm," IFAC-PapersOnLine, Vol. 51, No. 17, pp. 431-436, 2018. https://doi.org/10.1016/j.ifacol.2018.08.183.
- L. F. S. Pereira, S. Barbon, N. A. Valous, and D. F. Barbin, Predicting the ripening of papaya fruit with digital imaging and random forests, Computers and Electronics in Agriculture, Vol. 145, pp. 76-82, 2018. https://doi.org/10.1016/j.compag.2017.12.029.
- T. Anandhakrishnan, S.M. Jaisakthi, Deep Convolutional Neural Networks for image based tomato leaf disease detection, Sustainable Chemistry and Pharmacy, Vol. 30, p. 100793, 2022. https://doi.org/10.1016/j.scp.2022.100793.
- M. Zaborowicz, P. Boniecki, K. Koszela, A. Przybylak, and J. Przybył, Application of neural image analysis in evaluating the quality of greenhouse tomatoes, Scientia Horticulturae, Vol. 218, pp. 222-229, 2017, https://doi.org/10.1016/j.scienta.2017.02.001.
- T. Zeng, S. Li, Q. Song, F. Zhong, and X. Wei, Lightweight tomato real-time detection method based on improved YOLO and mobile deployment, Computers and Electronics in Agriculture, Vol. 205, p. 107625, 2023. https://doi.org/10.1016/j.compag.2023.107625.
- J. Qi, X. Liu, K. Liu, F. Xu, H. Guo, X. Tian, M. Li, Z. Bao, Y. Li, An improved YOLOv5 model based on visual attention mechanism: Application to recognition of tomato virus disease, Computers and Electronics in Agriculture, Vol. 194, p. 106780, 2022. https://doi.org/10.1016/j.compag.2022.106780.
- Q. Rong, C. Hu, X. Hu, M. Xu, Picking point recognition for ripe tomatoes using semantic segmentation and morphological processing, Computers and Electronics in Agriculture, Vol. 210, p. 107923, 2023. https://doi.org/10.1016/j.compag.2023.107923.
References
M. Stoelen et al., "Low-Cost Robotics for Horticulture: A Case Study on Automated Sugar Pea Harvesting," 10th European Conference on Precision Agriculture (ECPA), 2015. https://doi.org/10.3920/978-90-8686-814-8
T. Dewi, S. Nurmaini, P., Risma, and Y. Oktarina, Y., “Inverse Kinematic Analysis of 4 DOF Pick and Place Arm Robot Manipulator using Fuzzy Logic Controller,” International Journal of Electrical and Computer Engineering (IJECE), Vol. 10, No 2, pp. 1376-1386, 2020. http://doi.org/10.11591/ijece.v10i2.pp1376-1386
T. Dewi, C. Anggraini, P. Risma, Y. Oktarina, and Muslikhin, “Motion Control Analysis of Two Collaborative Arm Robots in Fruit Packaging System,” SINERGIA Vol. 25, No. 2, pp. 217-226, 2021. http://doi.org/10.22441/sinergi.2021.2.013
T. Dewi, Z. Mulya, P. Risma, and Y. Oktarina, “BLOB Analysis of an Automatic Vision Guided System for a Fruit Picking and Placing Robot,” International Journal of Computational Vision and Robotics, Vol. 11, No 3, pp. 315-326, 2021. https://doi.org/10.1504/IJCVR.2021.115161
C. Wang, Y. Tang, X. Zou, W. SiTu, and W. Feng, "A robust fruit image segmentation algorithm against varying illumination for vision system of fruit harvesting robot," Optik, Vol. 131, pp. 626-631, 2017. https://doi.org/10.1016/j.ijleo.2016.11.177.
C.W. Bac, J. Hemming, and E.J. Van Henten, “Robust pixel-based classification of obstacles for robotic harvesting of sweet-pepper,” Computers and Electronics in Agriculture, Vol. 96, pp. 148-162, 2013. https://doi.org/10.1016/j.compag.2013.05.004
N. M. Syahrian, P. Risma, and T. Dewi, “Vision-Based Pipe Monitoring Robot for Crack Detection using Canny Edge Detection Method as an Image Processing Technique,” Kinetik: Game Technology, Information System, Computer Network, Computing Electronics, and Control, Vol. 2, No. 4, pp. 243-250, 2017. https://doi.org/10.22219/kinetik.v2i4.243
M.D. Yusuf, RD. Kusumanto, Y. Oktarina, T. Dewi, and P. Risma, “Blob Analysis for Fruit Recognition and Detection,” Computer Engineering and Applications, Vol 7 No 1 pp. 23-32, 2018. https://doi.org/10.18495/comengapp.v7i1.237
T. Dewi, P. Risma, Y. Oktarina and S. Muslimin, "Visual Servoing Design and Control for Agriculture Robot; a Review," 2018 International Conference on Electrical Engineering and Computer Science (ICECOS), Pangkal, Indonesia, 2018, pp. 57-62. https://doi.org/10.1109/ICECOS.2018.8605209
H. Gharakhani, J. A. Thomasson, and Y. Lu, "An end-effector for robotic cotton harvesting," Smart Agricultural Technology, Vol. 2, p. 100043, 2022. https://doi.org/10.1016/j.atech.2022.100043.
T. Dewi, P. Risma, Y. Oktarina, and M. Nawawi, “Tomato Harvesting Arm Robot Manipulator; a Pilot Project,” Journal of Physics: Conference Series, 1500, p 012003, Proc. 3rd FIRST, Palembang: Indonesia, 2020. https://doi.org/10.1088/1742-6596/1500/1/012003
Z. Hou, Z. Li, T. Fadiji, and J. Fu,Soft, “Grasping Mechanism of Human Fingers for Tomato-picking Bionic Robots,” Computers and Electronics in Agriculture, Vol 182, 106010, 2021. https://doi.org/10.1016/j.compag.2021.106010
J. Chen, H. Qiang, J. Wu, G. Xu, and Z. Wang, “Navigation Path Extraction for Greenhouse Cucumber-picking Robots Using the Prediction-point Hough Transform, Computers and Electronics in Agriculture,” Vol. 180, 105911, 2021. https://doi.org/10.1016/j.compag.2020.105911
L. van Herck, P. Kurtser, L. Wittemans, and Y. Edan, “Crop Design for Improved Robotic Harvesting: a Case Study of Sweet Pepper Harvesting, Biosystems Engineering,” Vol 192, pp. 294-308, 2020. https://doi.org/10.1016/j.biosystemseng.2020.01.021
Y. Zhao, L. Gong, C. Liu, and Y. Huang, "Dual-arm Robot Design and Testing for Harvesting Tomato in Greenhouse," IFAC-PapersOnLine, Vol. 49, No 16, pp. 161-165, 2016. https://doi.org/10.1016/j.ifacol.2016.10.030
T. Dewi, P. Risma, and Y. Oktarina, “Fruit Sorting Robot based on Color and Size for an Agricultural Product Packaging System,” Bulletin of Electrical Engineering, and Informatics (BEEI), vol. 9, no. 4, pp. 1438-1445, 2020. https://doi.org/10.11591/eei.v9i4.2353
A. Nasiri, A. Taheri-Garavand, and Y. Zhang Image-based deep learning automated sorting of date fruit, Postharvest Biology and Technology, Vol. 153, pp. 133-141, 2019. https://doi.org/10.1016/j.postharvbio.2019.04.003.
M. Fashi, L. Naderloo, and H. Javadikia, The relationship between the appearance of pomegranate fruit and color and size of arils based on image processing, Postharvest Biology and Technology, Vol. 154, pp. 52-57, 2019. https://doi.org/10.1016/j.postharvbio.2019.04.017.
J. Jhawar, “Orange Sorting by Applying Pattern Recognition on Colour Image,” Procedia Computer Science, vol. 78, pp. 691–697, December 2016. https://doi.org/10.1016/j.procs.2016.02.118
L. Fu, J. Duan, X. Zou, G. Lin, S. Song, B. Ji, and Z. Yang, Banana detection based on color and texture features in the natural environment, Computers and Electronics in Agriculture, Vol. 167, p. 105057, 2019. https://doi.org/10.1016/j.compag.2019.105057.
U. Dorj, M. Lee, and S. Yun, An yield estimation in citrus orchards via fruit detection and counting using image processing, Computers and Electronics in Agriculture, Vol. 140, pp. 103-112, 2017. https://doi.org/10.1016/j.compag.2017.05.019.
L. Fu, Z. Liu, Y. Majeed, and Y. Cui, “Kiwifruit Yield Estimation using Processing by an Android Mobile Phone,” IFAC Conference Paper Archive, vol. 51, pp. 185–190, 2018. https://doi.org/10.1016/j.ifacol.2018.08.137
K. Tan, W. Suk, H. Gan, and S. Wang, “Recognising Blueberry Fruit of Different Maturity Using Histogram Oriented Gradients and Colour Features in Outdoor Scenes,” Biosystems Engineering, vol. 176, pp. 59–72, 2018. https://doi.org/10.1016/j.biosystemseng.2018.08.011
A. Septiarini, H. Hamdani, H. R. Hatta, and K. Anwar, "Automatic Image Segmentation of Oil Palm Fruits by Applying the Contour-Based Approach," Scientia Horticulturae, 2019. https://doi.org/10.1016/j.scienta.2019.108939.
M. H. Malik, T. Zhang, H. Li, M. Zhang, S. Shabbir, and A. Saeed, "Mature Tomato Fruit Detection Algorithm Based on improved HSV and Watershed Algorithm," IFAC-PapersOnLine, Vol. 51, No. 17, pp. 431-436, 2018. https://doi.org/10.1016/j.ifacol.2018.08.183.
L. F. S. Pereira, S. Barbon, N. A. Valous, and D. F. Barbin, Predicting the ripening of papaya fruit with digital imaging and random forests, Computers and Electronics in Agriculture, Vol. 145, pp. 76-82, 2018. https://doi.org/10.1016/j.compag.2017.12.029.
T. Anandhakrishnan, S.M. Jaisakthi, Deep Convolutional Neural Networks for image based tomato leaf disease detection, Sustainable Chemistry and Pharmacy, Vol. 30, p. 100793, 2022. https://doi.org/10.1016/j.scp.2022.100793.
M. Zaborowicz, P. Boniecki, K. Koszela, A. Przybylak, and J. Przybył, Application of neural image analysis in evaluating the quality of greenhouse tomatoes, Scientia Horticulturae, Vol. 218, pp. 222-229, 2017, https://doi.org/10.1016/j.scienta.2017.02.001.
T. Zeng, S. Li, Q. Song, F. Zhong, and X. Wei, Lightweight tomato real-time detection method based on improved YOLO and mobile deployment, Computers and Electronics in Agriculture, Vol. 205, p. 107625, 2023. https://doi.org/10.1016/j.compag.2023.107625.
J. Qi, X. Liu, K. Liu, F. Xu, H. Guo, X. Tian, M. Li, Z. Bao, Y. Li, An improved YOLOv5 model based on visual attention mechanism: Application to recognition of tomato virus disease, Computers and Electronics in Agriculture, Vol. 194, p. 106780, 2022. https://doi.org/10.1016/j.compag.2022.106780.
Q. Rong, C. Hu, X. Hu, M. Xu, Picking point recognition for ripe tomatoes using semantic segmentation and morphological processing, Computers and Electronics in Agriculture, Vol. 210, p. 107923, 2023. https://doi.org/10.1016/j.compag.2023.107923.