A Deep-Learning Classification Framework for Reducing Communication Errors in Dynamic Hand Signaling for Crane Operation
Publication: Journal of Construction Engineering and Management
Volume 149, Issue 2
Abstract
Crane operators and signalmen play an integral role in the safe and efficient operation of cranes on a construction site. Operating a crane is a complex and demanding task that requires careful coordination between operator and signalmen in order to avoid errors that could have dire consequences, including serious injury or loss of life. Therefore, special considerations should be taken to mitigate communication errors that could occur between the two parties. Technology can play an important role in enhancing communication, and, with recent advancements in technology, human–computer interaction has emerged as an active area of research within the field of computer vision. This paper presents a framework that integrates the YOLOv4 model (for object detection) and the long short-term memory (LSTM) model (a recurrent neural network) for dynamic hand signal classification in real time. The first step is the creation of a crane signalman dynamic hand signal data set with 18 classes. The YOLOv4 model is then customized for this application by modifying the activation function. Three modified YOLOv4 models are then integrated with the LSTM model. The modified YOLOv4 integrated with LSTM is found to achieve a maximum overall accuracy of 94.8% with an inference time of 55.1 frames per second. The model is further validated with real-time dynamic hand signal classification, achieving an accuracy of 93.5% and an inference time of 44 frames per second. The proposed models show improved quality in classification accuracy as well as in processing speed in comparison to some of the most widely used models currently in use. The proposed novel framework can be used as another layer of communication to supplement current practice and reduce communication errors between crane signalmen and crane operators.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
References
Agarap, A. F. M. 2018. “Deep learning using rectified linear units (ReLU).” Preprint, submitted March 22, 2018. https://arxiv.org/abs/1803.08375.
Bae, J. H., N. T. Le, and J. T. Kim. 2017. “Smartphone image receiver architecture for optical camera communication.” Wireless Pers. Commun. 93 (4): 1043–1066. https://doi.org/10.1007/s11277-017-3971-3.
Beavers, J. E., F. Asce, J. R. Moore, R. Rinehart, and W. R. Schriver. 2006. “Crane-related fatalities in the construction industry.” J. Constr. Eng. Manage. 132 (9): 901–910. https://doi.org/10.1061/(ASCE)0733-9364(2006)132:9(901).
Bochkovskiy, A., C. Wang, and H. M. Liao. 2020. “YOLOv4: Optimal speed and accuracy of object detection.” Preprint, submitted April 23, 2020. https://arxiv.org/abs/2004.10934.
Boncelet, C. 2009. Image noise models: The essential guide to image processing. 1st ed. Amsterdam, Netherlands: Elsevier.
Bull, D. 2014. “Digital picture formats and representation.” In Communicating pictures, 99–132. Amsterdam, Netherlands: Elsevier.
Bust, P. D., A. G. F. Gibb, and S. Pink. 2008. “Managing construction health and safety: Migrant workers and communicating safety messages.” Saf. Sci. 46 (4): 585–602. https://doi.org/10.1016/j.ssci.2007.06.026.
Chen, Y., H. Chi, S. Kangm, and S. Hsieh. 2011. “A smart crane operations assistance system using augmented reality technology.” In Proc., 28th Int. Symp. on Automation and Robotics in Construction, ISARC 2011, 643–649. Seoul: International Association for Automation and Robotics in Construction.
Everett, B. J. G., and A. H. Slocum. 1993. “Device for improving crane.” J. Constr. Eng. Manage. 119 (1): 23–39. https://doi.org/10.1061/(ASCE)0733-9364(1993)119:1(23).
Fang, Q., H. Li, X. Luo, L. Ding, H. Luo, and C. Li. 2018a. “Computer vision aided inspection on falling prevention measures for steeplejacks in an aerial environment.” Autom. Constr. 93 (Sep): 148–164. https://doi.org/10.1016/j.autcon.2018.05.022.
Fang, Y., and Y. K. Cho. 2016. “A framework of lift virtual prototyping (LVP) approach for crane safety planning.” In Proc., 33rd Int. Symp. on Automation and Robotics in Construction, ISARC 2016, 291–297. Auburn, AL: International Association for Automation and Robotics in Construction.
Fang, Y., Y. K. Cho, F. Druso, and J. Seo. 2018b. “Assessment of operator’s situation awareness for smart operation of mobile cranes.” Autom. Constr. 85 (Jan): 65–75. https://doi.org/10.1016/j.autcon.2017.10.007.
Ghiasi, G., T. Lin, and Q. V. Le. 2018. “DropBlock: A regularization method for convolutional networks.” In Advances in neural information processing systems, 1–11. Pasadena, CA: Neural Information Processing Systems.
Greff, K., R. K. Srivastava, J. Koutník, B. R. Steunebrink, and J. Schmidhuber. 2016. “LSTM: A search space odyssey.” IEEE Trans. Neural Networks Learn. Syst. 28 (10): 2222–2232. https://doi.org/10.1109/TNNLS.2016.2582924.
Hernandez-Garcia, A., and P. Konig. 2018. “Further advantages of data augmentation on convolutional neural networks.” In Proc., Int. Conf. on Artificial Neural Networks, 95–103. New York: Springer.
Hou, X., Y. Zhang, and J. Hou. 2020. “Application of YOLO V2 in construction vehicle detection.” In Proc., Int. Conf. on Natural Computation, Fuzzy Systems and Knowledge Discovery, 1249–1256. New York: Springer.
Hu, J., X. Geo, H. Wu, and S. Gao. 2019. “Detection of workers without the helments in videos based on YOLO V3.” In Proc., 12th Int. Congress on Image and Signal Processing, BioMedical Engineering and Informatics, 1553–1560. New York: IEEE.
Huang, J., W. Zhou, Q. Zhang, H. Li, and W. Li. 2018. “Video-based sign language recognition without temporal segmentation.” In Vol. 32 of Proc., AAAI Conf. on Artificial Intelligence. Vancouver, BC, Canada: Public Knowledge Project.
Huang, Z., J. Wang, X. Fu, T. Yu, Y. Guo, and R. Wang. 2020. “DC-SPP-YOLO: Dense connection and spatial pyramid pooling based YOLO for object detection.” Inf. Sci. 522 (Jun): 241–258. https://doi.org/10.1016/j.ins.2020.02.067.
Kim, J. A., J. Y. Sung, and S. H. Park. 2020. “Comparison of Faster-RCNN, YOLO, and SSD for real-time vehicle type recognition.” In Proc., 2020 IEEE Int. Conf. on Consumer Electronics-Asia (ICCE-Asia), 1–4. New York: IEEE.
Kines, P., L. P. S. Andersen, S. Spangenberg, K. L. Mikkelsen, J. Dyreborg, and D. Zohar. 2010. “Improving construction site safety through leader-based verbal safety communication.” J. Saf. Res. 41 (5): 399–406. https://doi.org/10.1016/j.jsr.2010.06.005.
King, R. A. 2012. “Analysis of crane and lifting accidents in North America from 2004 to 2010.” Ph.D. dissertation, Dept. of Civil and Environmental Engineering, Massachusetts Institute of Technology.
Kukkala, V. K., J. Tunnell, S. Pasricha, and T. Bradley. 2018. “Advanced driver-assistance systems: A path toward autonomous vehicles.” IEEE Consum. Electron. Mag. 7 (5): 18–25. https://doi.org/10.1109/MCE.2018.2828440.
Liu, S., L. Qi, H. Qin, J. Shi, and J. Jia. 2018. “Path aggregation network for instance segmentation.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 8759–8768. New York: IEEE.
Mansoor, A., S. Liu, G. M. Ali, A. Bouferguene, and M. Al-Hussein. 2020. “Conceptual framework for safety improvement in mobile cranes.” In Proc., Construction Research Congress 2020: Computer Applications, 964–971. Reston, VA: ASCE.
Mikołajczyk, A., and M. Grochowski. 2018. “Data augmentation for improving deep learning in image classification problem.” In Proc., Int. Interdisciplinary PhD Workshop (IIPhDW), 117–122. New York: IEEE.
Misra, D. 2019. “MISH: A self regularized non-monotonic activation function.” Preprint, submitted August 23, 2019. https://arxiv.org/abs/1908.08681.
Misra, S., and Y. Wu. 2019. “Machine learning assisted segmentation of scanning electron microscopy images of organic-rich shales with feature extraction and feature ranking.” Mach. Learn. Subsurface Charact. 289 (Jan): 289–313. https://doi.org/10.1016/b978-0-12-817736-5.00010-7.
Molchanov, P., X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz. 2016. “Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 4207–4215. New York: IEEE.
Neitzel, R. L., N. S. Seixas, and K. K. Ren. 2001. “A review of crane safety in the construction industry.” Appl. Occup. Environ. Hyg. 16 (12): 1106–1117. https://doi.org/10.1080/10473220127411.
Nepal, U., and H. Eslamiat. 2022. “Comparing YOLOv3, YOLOv4 and YOLOv5 for autonomous landing spot detection in faulty UAVs.” Sensors 22 (2): 464. https://doi.org/10.3390/s22020464.
Okan, K., G. Ahmet, K. Neslihan, and R. Gerhard. 2019. “Real-time hand gesture detection and classification using convolutional neural networks.” In Proc., 14th IEEE Int. Conf. on Automatic Face & Gesture Recognition, 1–8. New York: IEEE.
OSHA (Occupational Safety and Health Administration). 2021. “Cranes and derricks in construction.” Accessed April 19, 2020. https://open.alberta.ca/dataset/757fed78-8793-40bb-a920-6f000853172b/resource/9296e033-fd12-40dc-ac86-21e5873d4161/download/4403880-part-6-cranes-hoists-and-lifting-devices.pdf.
Oyedotun, O. K., and A. Khashman. 2017. “Deep learning in vision-based static hand gesture recognition.” Neural Comput. Appl. 28 (12): 3941–3951. https://doi.org/10.1007/s00521-016-2294-8.
Park, K., H. Lee, H. Kim, J. I. Kim, H. Lee, and M. W. Pyeon. 2011. “AR-HUD system for tower crane on construction field.” In Proc., 2011 IEEE Int. Symp. on Virtual Reality Innovation, 261–266. New York: IEEE.
Pickering, C. A., K. J. Burnham, and M. J. Richardson. 2007. “A research study of hand gesture recognition technologies and applications for human vehicle interaction.” In Proc., 3rd Institution of Engineering and Technology Conf. on Automotive Electronics, 1–15. New York: IEEE.
Potter, M. C., B. Wyble, C. E. Hagmann, and E. S. McCourt. 2014. “Detecting meaning in RSVP at 13 ms per picture.” Attention Percept. Psychophysics 76 (2): 270–279. https://doi.org/10.3758/s13414-013-0605-z.
Qasim, A. B., and A. Pettirsch. 2020. “Recurrent neural networks for video object detection.” Preprint, submitted October 29, 2020. https://arxiv.org/abs/2010.15740.
Rahman, E. U., Y. Zhang, S. Ahmad, H. I. Ahmad, and S. Jobaer. 2021. “Autonomous vision-based primary distribution systems porcelain insulators inspection using UAVs.” Sensors 21 (3): 974. https://doi.org/10.3390/s21030974.
Ramachandran, P., B. Zoph, and Q. V. Le. 2017. “Searching for activation functions.” Preprint, submitted October 16, 2017. https://arxiv.org/abs/1710.05941.
Raviv, G., and A. Shapira. 2018. “Systematic approach to crane-related near-miss analysis in the construction industry.” Int. J. Construct. Manage. 18 (4): 310–320. https://doi.org/10.1080/15623599.2017.1382067.
Shapira, A., Y. Rosenfeld, and I. Mizrahi. 2008. “Vision system for tower cranes.” J. Constr. Eng. Manage. 134 (5): 320–332. https://doi.org/10.1061/(ASCE)0733-9364(2008)134:5(320).
Su, H., W. Qi, C. Yang, J. Sandoval, G. Ferrigno, and E. De Momi. 2020. “Deep neural network approach in robot tool dynamics identification for bilateral teleoperation.” IEEE Rob. Autom. Lett. 5 (2): 2943–2949. https://doi.org/10.1109/LRA.2020.2974445.
US Bureau of Labor Statistics. 2017. “Fatal occupational injuries involving cranes.” Accessed September 13, 2021. https://www.bls.gov/iif/oshwc/cfoi/cranes-2017.htm.
US Dept. of Labor. 2003. “Census of fatal occupational injuries summary, 2003.” Accessed January 19, 2022. https://stats.bls.gov/news.release/cfoi.nr0.htm.
Xu, J., Z. Li, B. Du, M. Zhang, and J. Liu. 2020. “Reluplex made more practical: Leaky ReLU.” In Proc., 2020 IEEE Symp. on Computers and Communications, 1–7. New York: IEEE.
Yin, X., Y. Chen, A. Bouferguene, H. Zaman, M. Al-Hussein, and L. Kurach. 2020. “A deep learning-based framework for an automated defect detection system for sewer pipes.” Autom. Constr. 109 (Jan): 102967. https://doi.org/10.1016/j.autcon.2019.102967.
Zavichi, A., and A. H. Behzadan. 2011. “A real time decision support system for enhanced crane operations in construction and manufacturing.” In Computing in civil engineering, 586–593. Reston, VA: ASCE.
Zekavat, P. R., and L. Bernold. 2014. “Embedded wireless communication platform addresses crane safety and efficiency.” In Proc., Construction Research Congress 2014, 309–318. Reston, VA: ASCE.
Zekavat, P. R., S. Moon, and L. E. Bernold. 2015. “Holonic construction management: Unified framework for ICT-supported process control.” J. Manage. Eng. 31 (1): A4014008 . https://doi.org/10.1061/(ASCE)ME.1943-5479.0000316.
Zhao, Q. 2011. “Cause analysis of U.S. crane-related accidents.” Ph.D. dissertation, Dept. of Building Construction, Univ. of Florida.
Zheng, Z., P. Wang, W. Liu, J. Li, R. Ye, and D. Ren. 2019. “Distance-IoU loss: Faster and better learning for bounding box regression.” In Proc., AAAI Conf. on Artificial Intelligence, 12993–13000. Vancouver, BC, Canada: Public Knowledge Project.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 149 • Issue 2 • February 2023
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Jun 18, 2022
Accepted: Sep 28, 2022
Published online: Dec 1, 2022
Published in print: Feb 1, 2023
Discussion open until: May 1, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Asif Mansoor, Shuai Liu, Ahmed Bouferguene, Mohamed Al-Hussein, Crane Signalman Hand-Signal Classification Framework Using Sensor-Based Smart Construction Glove and Machine-Learning Algorithms, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-14458, 150, 8, (2024).
- Seunghyeon Wang, Mincheol Kim, Hyeonyong Hae, Mengqiu Cao, Juhyung Kim, The Development of a Rebar-Counting Model for Reinforced Concrete Columns: Using an Unmanned Aerial Vehicle and Deep-Learning Approach, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-13686, 149, 11, (2023).
- Asif Mansoor, Shuai Liu, Ghulam Muhammad Ali, Ahmed Bouferguene, Mohamed Al-Hussein, Scientometric analysis and critical review on the application of deep learning in the construction industry, Canadian Journal of Civil Engineering, 10.1139/cjce-2022-0379, (2022).