Crane Signalman Hand-Signal Classification Framework Using Sensor-Based Smart Construction Glove and Machine-Learning Algorithms
Publication: Journal of Construction Engineering and Management
Volume 150, Issue 8
Abstract
On construction sites, the principal means of communication between crane operator and crane signalman is mainly hand signaling. Often these hand signals are received and interpreted incorrectly, leading to communication errors and accidents. This paper describes the development of a sensor-based smart construction glove that uses a 3-axis accelerometer, 3-axis gyroscope, and 3-axis magnetometer to measure the hand’s orientation with the help of quaternions, and flex-sensors to measure the intensity of bend in the fingers. The noises in the sensors are removed using a complementary filter fusion algorithm. Four machine-learning models—-nearest neighbor, support vector machine, decision tree, and convolutional neural network–long short-term memory (CNN-LSTM)—are proposed for crane signalman hand-signal classification. The models are trained, validated, and tested using the sensor data collected from the smart construction glove. The best performance in the test data set is achieved by CNN-LSTM, which is found to achieve a precision of 84.3%, recall of 83.9%, an F1-score of 84%, and an average accuracy of 94.22% in the test data set. For real-time crane signalman hand-signal classification, an Android-based mobile application is developed. The application receives the data in text format from the smart construction glove via Bluetooth and converts it into speech output. The smart construction glove can classify 18 different crane signalman hand signals used on construction sites. The CNN-LSTM model is found to achieve the highest overall accuracy (93.87%) in real-time implementation.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
References
Agarap, A. F. M. 2018. “Deep learning using rectified linear units (ReLU).” Preprint, submitted March 22, 2018. http://arxiv.org/abs/1803.08375.
Ahn, C. R., S. Lee, C. Sun, H. Jebelli, K. Yang, and B. Choi. 2019. “Wearable sensing technology applications in construction safety and health.” J. Constr. Eng. Manage. 145 (11): 03119007. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001708.
Akhavian, R., and A. H. Behzadan. 2016. “Smartphone-based construction workers’ activity recognition and classification.” Autom. Constr. 71 (2): 198–209. https://doi.org/10.1016/j.autcon.2016.08.015.
Al Mamun, A., M. S. J. K. Polash, and F. M. Alamgir. 2017. “Flex sensor based hand glove for deaf and mute people.” Int. J. Comput. Networks Commun. Secur. 5 (2): 38.
ASME. 2021. “B30.5-2021: Mobile and locomotive cranes.” Accessed April 22, 2024. https://www.asme.org/codes-standards/find-codes-standards/b30-5-mobile-locomotive-cranes.
Awolusi, I., E. Marks, and M. Hallowell. 2018. “Wearable technology for personalized construction safety monitoring and trending: Review of applicable devices.” Autom. Constr. 85 (Jan): 96–106. https://doi.org/10.1016/j.autcon.2017.10.010.
Begg, R. K., M. Palaniswami, and B. Owen. 2005. “Support vector machines for automated gait classification.” IEEE Trans. Biomed. Eng. 52 (5): 828–838. https://doi.org/10.1109/TBME.2005.845241.
Bonato, P. 2009. “Advances in wearable technology for rehabilitation.” In Advanced technologies in rehabilitation, 145–159. Amsterdam, Netherlands: IOS Press.
Bureau of Labor Statistics. 2017. “Fatal occupational injuries involving cranes.” Accessed February 13, 2021. https://www.bls.gov/iif/oshwc/cfoi/cranes-2017htm#:∼:text=From%202011%20to%202017%2C%20the,297%20fatal%20injuries%20involving%20cranes.
Chaudhury, S. B., M. Sengupta, and K. Mukherjee. 2014. “Vibration monitoring of rotating machines using MEMS accelerometer.” Int. J. Sci. Eng. Res. 2 (9): 5–11.
Chuang, W. C., W. J. Hwang, T. M. Tai, D. R. Huang, and Y. J. Jhang. 2019. “Continuous finger gesture recognition based on flex sensors.” Sensors 19 (18): 1–11. https://doi.org/10.1109/JSEN.2019.2920795.
Dong, W., L. Yang, and G. Fortino. 2020. “Stretchable human machine interface based on smart glove embedded with PDMS-CB strain sensors.” IEEE Sens. J. 20 (14): 8073–8081. https://doi.org/10.1109/JSEN.2020.2982070.
Fan, L., Z. Wang, and H. Wang. 2013. “Human activity recognition model based on decision tree.” In Proc., Int. Conf. on Advanced Cloud and Big Data, 64–68. New York: IEEE.
Fang, Q., H. Li, X. Luo, L. Ding, H. Luo, and C. Li. 2018a. “Computer vision aided inspection on falling prevention measures for steeplejacks in an aerial environment.” Autom. Constr. 93 (Sep): 148–164. https://doi.org/10.1016/j.autcon.2018.05.022.
Fang, Y., Y. K. Cho, F. Druso, and J. Seo. 2018b. “Assessment of operator’s situation awareness for smart operation of mobile cranes.” Autom. Constr. 85 (Jan): 65–75. https://doi.org/10.1016/j.autcon.2017.10.007.
Ghiasi, M. M., S. Zendehboudi, and A. A. Mohsenipour. 2020. “Decision tree-based diagnosis of coronary artery disease: CART model.” Comput. Methods Programs Biomed. 192 (Aug): 105400. https://doi.org/10.1016/j.cmpb.2020.105400.
Gupta, H. P., H. S. Chudgar, S. Mukherjee, T. Dutta, and K. Sharma. 2016. “A continuous hand gestures recognition technique for human-machine interaction using accelerometer and gyroscope sensors.” IEEE Sens. J. 16 (16): 6425–6432. https://doi.org/10.1109/JSEN.2016.2581023.
Hagan, P. E., J. F. Montgomery, and J. T. O’Reilly. 2015. Accident prevention manual for business & industry: Engineering & technology. Chicago: National Safety Council.
Hemingway, E. G., and O. M. O’Reilly. 2018. “Perspectives on Euler angle singularities, gimbal lock, and the orthogonality of applied forces and applied moments.” Multibody Syst. Dyn. 44 (1): 31–56. https://doi.org/10.1007/s11044-018-9620-0.
Hwag, S., J. Seo, H. Jebelli, and S. Lee. 2016. “Feasibility analysis of heart rate monitoring of construction workers using a photoplethysmography (PPG) sensor embedded in a wristband-type activity tracker.” Autom. Constr. 71 (Nov): 372–381. https://doi.org/10.1016/j.autcon.2016.08.029.
Jani, A. B., N. A. Kotak, and A. K. Roy. 2018. “Sensor based hand gesture recognition system for English alphabets used in sign language of deaf-mute people.” In Proc., IEEE Sensors, 17–20. New York: IEEE.
Jebelli, H., B. Choi, and S. Lee. 2019. “Application of wearable biosensors to construction sites. I: Assessing workers’ stress.” J. Constr. Eng. Manage. 145 (12): 04019079. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001729.
Jo, B. W., Y. S. Lee, R. M. A. Khan, J. H. Kim, and D. K. Kim. 2019. “Robust construction safety system (RCSS) for collision accidents prevention on construction sites.” Sensors 19 (4): 932. https://doi.org/10.3390/s19040932.
Kaghyan, S., and H. Sarukhanyan. 2012. “Activity recognition using k-nearest neighbor algorithm on smartphone with tri-axial accelerometer.” Int. J. Inf. Models Anal. 1 (Jul): 146–156.
Kanan, R., O. Elhassan, and R. Bensalem. 2018. “An IoT-based autonomous system for workers’ safety in construction sites with real-time alarming, monitoring, and positioning strategies.” Autom. Constr. 88 (Apr): 73–86. https://doi.org/10.1016/j.autcon.2017.12.033.
Kim, J., A. S. Campbell, B. E. F. de Ávila, and J. Wang. 2019. “Wearable biosensors for healthcare monitoring.” Nat. Biotechnol. 37 (4): 389–406. https://doi.org/10.1038/s41587-019-0045-y.
Kines, P., L. P. S. Andersen, S. Spangenberg, K. L. Mikkelsen, J. Dyreborg, and D. Zohar. 2010. “Improving construction site safety through leader-based verbal safety communication.” J. Saf. Res. Nat. Saf. Council 41 (5): 399–406. https://doi.org/10.1016/j.jsr.2010.06.005.
Lai, X., T. Yang, Z. Wang, and P. Chen. 2019. “IoT implementation of Kalman Filter to improve accuracy of air quality monitoring and prediction.” Appl. Sci. 9 (9): 1831. https://doi.org/10.3390/app9091831.
Mandong, A., and U. Munir. 2018. “Smartphone based activity recognition using k-nearest neighbor algorithm.” In Proc., Int. Conf. on Engineering Technologies, 26–28. Konya, Turkey: Sanat Bilgi Teknolojileri.
Mansoor, A., S. Liu, G. M. Ali, A. Bouferguene, and M. Al-Hussein. 2020. “Conceptual framework for safety improvement in mobile cranes.” In Proc., Construction Research Congress: Computer Applications, 964–971. Reston, VA: ASCE.
Mansoor, A., S. Liu, G. M. Ali, A. Bouferguene, and M. Al-Hussein. 2023. “A deep-learning classification framework for reducing communication errors in dynamic hand signaling for crane operation.” J. Constr. Eng. Manage. 149 (2): 04022167. https://doi.org/10.1061/JCEMD4.COENG-12811.
Marks, E. D., and J. Teizer. 2013. “Method for testing proximity detection and alert technology for safe construction equipment operation.” Construct. Manage. Econ. 31 (6): 636–646. https://doi.org/10.1080/01446193.2013.783705.
McGinnis, R. S., S. M. Cain, S. P. Davidson, R. V. Vitali, S. G. McLean, and N. C. Perkins. 2014. “Validation of complementary filter based IMU data fusion for tracking torso angle and rifle orientation.” In Proc., ASME Int. Mechanical Engineering Congress and Exposition. New York: ASME.
Minh, V. T., R. Moezzi, and N. Katushin. 2019. “Haptic smart glove for augmented and virtual reality.” Sens. Lett. 17 (5): 358–364. https://doi.org/10.1166/sl.2019.4070.
Nath, N. D., R. Akhavian, and A. H. Behzadan. 2017. “Ergonomic analysis of construction worker’s body postures using wearable mobile sensors.” Appl. Ergon. 62 (Jul): 107–117. https://doi.org/10.1016/j.apergo.2017.02.007.
National Commission for the Certification of Crane Operators. 2014. Signalperson reference manual. Fairfax, VA: National Commission for the Certification of Crane Operators.
Neitzel, R. L., N. S. Seixas, and K. K. Ren. 2001. “A review of crane safety in the construction industry.” Appl. Occup. Environ. Hyg. 16 (12): 1106–1117. https://doi.org/10.1080/10473220127411.
Nonnarit, O., and A. Barreto. 2016. “Gyroscope drift correction algorithm for inertial measurement unit used in hand motion tracking.” In Proc., 2016 IEEE Sensors. New York: IEEE.
O’Flynn, B., J. Sachez-Torres, S. Tedesco, B. Downes, J. Connolly, J. Condell, and K. Curran. 2015. “Novel smart glove technology as a biomechanical monitoring tool.” Sens. Transducers 193 (10): 23–32.
Park, J., Y. K. Cho, and S. K. Timalsina. 2016. “Direction aware bluetooth low energy based proximity detection system for construction work zone safety.” In Proc., 33rd Int. Symp. on Automation and Robotics in Construction, 76–82. Edinburgh, UK: International Association for Automation and Robotics in Construction.
Pathak, V., S. Mongia, and G. Chitranshi. 2016. “A framework for hand gesture recognition based on fusion of Flex, Contact and accelerometer sensor.” In Proc., 3rd Int. Conf. on Image Information Processing, 312–319. New York: IEEE.
Raheja, J. L., R. Shyam, U. Kumar, and P. B. Prasad. 2010. “Real-time robotic hand control using hand gestures.” In Proc., 2nd Int. Conf. on Machine Learning and Computing, 12–16. New York: IEEE.
Raviv, G., and A. Shapira. 2018. “Systematic approach to crane-related near-miss analysis in the construction industry.” Int. J. Constr. Manage. 18 (4): 310–320. https://doi.org/10.1080/15623599.2017.1382067.
Roelofs, C., L. Sprague-Martinez, M. Brunette, and L. Azaroff. 2011. “A qualitative investigation of Hispanic construction worker perspectives on factors impacting worksite safety and risk.” Environ. Health 10 (Dec): 1–9. https://doi.org/10.1186/1476-069X-10-84.
Rosero-Montalvo, P. D., P. Godoy-Trujillo, E. Flores-Bosmediano, J. Carrascal-Garcia, S. Otero-Potosi, H. Benitez-Pereira, and D. H. Peluffo-Ordonez. 2018. “Sign language recognition based on intelligent glove using machine learning techniques.” In Proc., IEEE 3rd Ecuador Technical Chapters Meeting, 5–9. New York: IEEE.
Saggio, G. 2012. “Mechanical model of flex sensors used to sense finger movements.” Sens. Actuators A 185 (Apr): 53–58. https://doi.org/10.1016/j.sna.2012.07.023.
Sathiyanarayanan, M., S. Azharuddin, S. Kumar, and G. Khan. 2014. “Gesture controlled robot for military purpose.” Int. J. Technol. Res. Eng. 1 (11): 1300–1303.
Shibuya, N., B. T. Nukala, A. I. Rodriguez, J. Tsay, T. Q. Nguyen, S. Zupancic, and D. Y. Lie. 2015. “A real-time fall detection system using a wearable gait analysis sensor and a Support Vector Machine (SVM) classifier.” In Proc., 8th Int. Conf. on Mobile Computing and Ubiquitous Networking. 66–67. New York: IEEE.
Sriram, N., and M. Nithiyanandham. 2013. “A hand gesture recognition based communication system for silent speakers.” In Proc., Int. Conf. on Human Computer Interactions. New York: IEEE.
Taneja, S., and M. Sukhija. 2020. “Technical paper gift of voice to mute: Hand gestures converted to text and voice.” Int. J. Comput. Sci. Eng. 8 (4): 64–69.
Tu, D., D. Bein, and M. Gofman. 2020. “Designing a unity game using the haptic feedback gloves, VMG 30 Plus.” In Proc., 17th Int. Conf. on Information Technology–New Generations (ITNG 2020), 393–400. Berlin: Springer.
Valenti, R. G., I. Dryanovski, and J. Xiao. 2015. “Keeping a good attitude: A quaternion-based orientation filter for IMUs and MARGs.” Sensors 15 (8): 19302–19330. https://doi.org/10.3390/s150819302.
Wang, X., and Z. Zhu. 2021. “Vision-based hand signal recognition in construction: A feasibility study.” Autom. Constr. 125 (14): 103625. https://doi.org/10.1016/j.autcon.2021.103625.
Zhang, F. 1997. “Quaternions and matrices of quaternions.” Linear Algebr. Appl. 251 (Jan): 21–57. https://doi.org/10.1016/0024-3795(95)00543-9.
Zhang, X., X. Chen, Y. Li, V. Lantz, K. Wang, and J. Yang. 2011. “A framework for hand gesture recognition based on accelerometer and EMG sensors.” IEEE Trans. Syst. Man Cyber. Part A:Syst. Humans 41 (6): 1064–1076. https://doi.org/10.1109/TSMCA.2011.2116004.
Information & Authors
Information
Published In
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Aug 29, 2023
Accepted: Feb 6, 2024
Published online: Jun 8, 2024
Published in print: Aug 1, 2024
Discussion open until: Nov 8, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.