Action Recognition Based on 3D Skeleton and LSTM for the Monitoring of Construction Workers’ Safety Harness Usage
Publication: Journal of Construction Engineering and Management
Volume 149, Issue 4
Abstract
Fall from height (FFH) is the most common construction accident in the construction industry, thus it is significant to monitor the use of safety harnesses, which are critical to the prevention of FFH. Sensing or computer vision technologies have been adopted to identify workers’ safety harness usage. However, previous research focused mainly on whether a worker wears a safety harness rather than on whether he or she properly fixes it to a lifeline, which is vital to prevent FFH but difficult to monitor. This research establishes an action recognition method based on a three-dimensional (3D) skeleton and long short-term memory (LSTM) to aid in automatically monitoring whether safety harnesses are fixed properly on site. An indoor experiment, which considered the features of a common real construction scenario—working on scaffolding—was conducted to test the effectiveness and feasibility of the proposed method. The result shows that the method achieves an acceptable precision and recall rate and can be used to detect the incorrect use of safety harnesses by combining multiple actions. This will contribute to the prevention of FFH in practice as well as to the body of knowledge of construction safety management.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request (data set and Python code for recognition model).
Acknowledgments
The authors thank the National Natural Science Foundation of China (Grant No. 51578318) and the Institute for Guo Qiang, Tsinghua University (Grant No. 2019GQC0004) for supporting this research.
References
Ale, B. J. M., L. J. Bellamy, H. Baksteen, M. Damen, L. H. J. Goossens, A. R. Hale, M. Mud, J. Oh, I. A. Papazoglou, and J. Y. Whiston. 2008. “Accidents in the construction industry in the Netherlands: An analysis of accident reports using Storybuilder.” Reliab. Eng. Syst. Saf. 93 (10): 1523–1533. https://doi.org/10.1016/j.ress.2007.09.004.
Borges, G. D., A. M. Reis, and A. R. P. Moro. 2019. “Ergonomic analysis of labor applied to scaffolders in a shipyard in Brazil.” In Proc., 20th Congress of the Int. Ergonomics Association (IEA 2018), 725–738. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-319-96089-0_79.
Cai, J., X. Li, X. Liang, and W. Wei. 2022. “Construction worker ergonomic assessment via LSTM-based multi-task learning framework.” In Proc., Construction Research Congress 2022, 215–224. Reston, VA: ASCE. https://doi.org/10.1061/9780784483961.023.
Carrara, F., P. Elias, J. Sedmidubsky, and P. Zezula. 2019. “LSTM-based real-time action detection and prediction in human motion streams.” Multimedia Tools Appl. 78 (19): 27309–27331. https://doi.org/10.1007/s11042-019-07827-3.
Chang, W.-T., C. J. Lin, Y.-H. Lee, and H.-J. Chen. 2018. “Development of an observational checklist for falling risk assessment of high-voltage transmission tower construction workers.” Int. J. Ind. Ergon. 68 (Nov): 73–81. https://doi.org/10.1016/j.ergon.2018.06.011.
Charreyron, S., S. Jackson, and L. F. Miranda-Moreno. 2013. “Toward a flexible system for pedestrian data collection with a Microsoft Kinect motion-sensing device.” Trans. Res. Rec. 2339 (1): 80–89. https://doi.org/10.3141/2339-09.
Chen, D., Y. Cai, J. Cui, J. Chen, H. Jiang, and M.-C. Huang. 2018. “Risk factors identification and visualization for work-related musculoskeletal disorders with wearable and connected gait analytics system and kinect skeleton models.” Smart Health 7–8 (Jun): 60–77. https://doi.org/10.1016/j.smhl.2018.05.003.
Cheung, K. M. G., S. Baker, and T. Kanade. 2003. “Shape-from-silhouette of articulated objects and its use for human body kinematics estimation and motion capture.” In Proc., 2003 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE. https://doi.org/10.1109/CVPR.2003.1211340.
Dai, C., X. Liu, and J. Lai. 2020. “Human action recognition using two-stream attention based LSTM networks.” Appl. Soft Comput. 86 (Jan): 105820. https://doi.org/10.1016/j.asoc.2019.105820.
Ding, L., W. Fang, H. Luo, P. E. D. Love, B. Zhong, and X. Ouyang. 2018. “A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory.” Autom. Constr. 86 (Feb): 118–124. https://doi.org/10.1016/j.autcon.2017.11.002.
Elias, P., J. Sedmidubsky, and P. Zezula. 2021. “Understanding the limits of 2D skeletons for action recognition.” Multimedia Syst. 27 (3): 547–561. https://doi.org/10.1007/s00530-021-00754-0.
Escorcia, V., M. A. Dávila, M. Golparvar-Fard, and J. C. Niebles. 2012. “Automated vision-based recognition of construction worker actions for building interior construction operations using RGBD cameras.” In Proc., Construction Research Congress 2012: Construction Challenges in a Flat World, 879–888. Reston, VA: ASCE. https://doi.org/10.1061/9780784412329.089.
Fang, W., L. Ding, H. Luo, and P. E. D. Love. 2018. “Falls from heights: A computer vision-based approach for safety harness detection.” Autom. Constr. 91 (Jul): 53–61. https://doi.org/10.1016/j.autcon.2018.02.018.
Fang, W., B. Zhong, N. Zhao, P. E. D. Love, H. Luo, J. Xue, and S. Xu. 2019. “A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network.” Adv. Eng. Inf. 39 (Jan): 170–177. https://doi.org/10.1016/j.aei.2018.12.005.
Ganguly, B., P. Vishwakarma, S. Biswas, and Rahul. 2020. Kinect sensor based single person hand gesture recognition for man–machine interaction. Singapore: Springer. https://doi.org/10.1007/978-981-13-8687-9_13.
Garcia-Garcia, A., S. Orts-Escolano, S. Oprea, V. Villena-Martinez, P. Martinez-Gonzalez, and J. Garcia-Rodriguez. 2018. “A survey on deep learning techniques for image and video semantic segmentation.” Appl. Soft Comput. 70 (Sep): 41–65. https://doi.org/10.1016/j.asoc.2018.05.018.
Gelmini, S., S. Strada, M. Tanelli, S. Savaresi, and A. Guzzon. 2020. “Automatic detection of human’s falls from heights for airbag deployment via inertial measurements.” Autom. Constr. 120 (Dec): 103358. https://doi.org/10.1016/j.autcon.2020.103358.
Golparvar-Fard, M., A. Heydarian, and J. C. Niebles. 2013. “Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers.” Adv. Eng. Inf. 27 (4): 652–663. https://doi.org/10.1016/j.aei.2013.09.001.
Gómez-de-Gabriel, J. M., J. A. Fernández-Madrigal, A. López-Arquillos, and J. C. Rubio-Romero. 2019. “Monitoring harness use in construction with BLE beacons.” Measurement 131 (Jan): 329–340. https://doi.org/10.1016/j.measurement.2018.07.093.
Grest, D., J. Woetzel, and R. Koch. 2005. “Nonlinear body pose estimation from depth images.” In Proc., Joint Pattern Recognition Symp., 285–292. Berlin: Springer. https://doi.org/10.1007/11550518_36.
Guo, H., Y. Yu, Q. Ding, and M. Skitmore. 2018. “Image-and-skeleton-based parameterized approach to real-time identification of construction workers’ unsafe behaviors.” J. Constr. Eng. Manage. 144 (6): 04018042. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001497.
Han, S., and S. Lee. 2013. “A vision-based motion capture and recognition framework for behavior-based safety management.” Autom. Constr. 35 (Nov): 131–141. https://doi.org/10.1016/j.autcon.2013.05.001.
Han, S., S. Lee, and F. Peña-Mora. 2011. “Application of dimension reduction techniques for motion recognition: Construction worker behavior monitoring.” In Proc., Int. Workshop on Computing in Civil Engineering, 102–109. Reston, VA: ASCE. https://doi.org/10.1061/41182(416)13.
Han, S., S. Lee, and F. Peña-Mora. 2014. “Comparative study of motion features for similarity-based modeling and classification of unsafe actions in construction.” J. Comput. Civ. Eng. 28 (5): A4014005. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000339.
Hasmori, M. F., N. A. F. Akhir, and I. Said. 2020. “Causes for lack of usage of satefy harness among construction workers in Malaysia: An investigation.” Recent Trends Civ. Eng. Built Environ. 1 (1): 104–114.
Hochreiter, S., and J. R. Schmidhuber. 1997. “Long short-term memory.” Neural Comput. 9 (8): 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735.
HSE (Health and Safety Executive). 2021. “The annual workplace fatality figures for 2020/21 released By The Health and Safety Executive (HSE).” Accessed February 5, 2023. https://www.hse-network.com/the-annual-workplace-fatality-figures-for-2020-21-released-by-the-health-and-safety-executive-hse/.
Huang, X., and J. Hinze. 2003. “Analysis of construction worker fall accidents.” J. Constr. Eng. Manage. 129 (3): 262–271. https://doi.org/10.1061/(ASCE)0733-9364(2003)129:3(262).
Ince, I. F. 2020. “Performance boosting of scale and rotation invariant human activity recognition (HAR) with LSTM networks using low dimensional 3D posture data in egocentric coordinates.” Appl. Sci. 10 (23): 8474. https://doi.org/10.3390/app10238474.
Jachowicz, M. 2015. “Corrosion of connectors used in equipment protecting against falls from a height.” Int. J. Occup. Saf. Ergon. 21 (3): 318–329. https://doi.org/10.1080/10803548.2015.1081771.
Johansson, G. 1973. “Visual perception of biological motion and a model for its analysis.” Percept. Psychophysics 14 (2): 201–211. https://doi.org/10.3758/BF03212378.
Joshua, L., and K. Varghese. 2011. “Accelerometer-based activity recognition in construction.” J. Comput. Civ. Eng. 25 (5): 370–379. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000097.
Jun, K., D.-W. Lee, K. Lee, S. Lee, and M. S. Kim. 2020. “Feature extraction using an RNN autoencoder for skeleton-based abnormal gait recognition.” IEEE Access 8 (Jan): 19196–19207. https://doi.org/10.1109/ACCESS.2020.2967845.
Kang, Y. 2018. “Use of fall protection in the US construction industry.” J. Manage. Eng. 34 (6): 04018045. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000655.
Kim, J. Y., and C. H. Caldas. 2013. “Vision-based action recognition in the internal construction site using interactions between worker actions and construction objects.” In Proc., Int. Symp. on Automation and Robotics in Construction. Oulu, Finland: International Association for Automation & Robotics in Construction. https://doi.org/10.22260/ISARC2013/0072.
Kim, K., and Y. K. Cho. 2020. “Effective inertial sensor quantity and locations on a body for deep learning-based worker’s motion recognition.” Autom. Constr. 113 (May): 103126. https://doi.org/10.1016/j.autcon.2020.103126.
Kong, T., W. Fang, P. E. D. Love, H. Luo, S. Xu, and H. Li. 2021. “Computer vision and long short-term memory: Learning to predict unsafe behaviour in construction.” Adv. Eng. Inf. 50 (Oct): 101400. https://doi.org/10.1016/j.aei.2021.101400.
Le, Q. T., A. Pedro, C. R. Lim, H. T. Park, C. S. Park, and H. K. Kim. 2015. “A framework for using mobile based virtual reality and augmented reality for experiential construction safety education.” Int. J. Eng. Educ. 31 (3): 713–725.
Lee, H., K. Yang, N. Kim, and C. R. Ahn. 2020. “A hybrid CNN-LSTM model for detecting excessive load carrying from workers’ body movements.” In Proc., Construction Research Congress 2020, 1137–1145. Reston, VA: ASCE. https://doi.org/10.1061/9780784482865.120.
Li, C., and S. Lee. 2011. “Computer vision techniques for worker motion analysis to reduce musculoskeletal disorders in construction.” In Proc., Int. Workshop on Computing in Civil Engineering. Reston, VA: ASCE. https://doi.org/10.1061/41182(416)47.
Li, C., P. Wang, S. Wang, Y. Hou, and W. Li. 2017a. “Skeleton-based action recognition using LSTM and CNN.” In Proc., 2017 IEEE Int. Conf. on Multimedia & Expo Workshops (ICMEW), 585–590. New York: IEEE. https://doi.org/10.1109/ICMEW.2017.8026287.
Li, C., Q. Zhong, D. Xie, and S. Pu. 2017b. “Skeleton-based action recognition with convolutional neural networks.” In Proc., 2017 IEEE Int. Conf. on Multimedia & Expo Workshops (ICMEW), 597–600. New York: IEEE. https://doi.org/10.1109/ICMEW.2017.8026285.
Li, Z., and D. Li. 2022. “Action recognition of construction workers under occlusion.” J. Build. Eng. 45 (Jan): 103352. https://doi.org/10.1016/j.jobe.2021.103352.
Lipscomb, H. J., A. M. Dale, V. Kaskutas, R. Sherman-Voellinger, and B. Evanoff. 2008. “Challenges in residential fall prevention: Insight from apprentice carpenters.” Am. J. Ind. Med. 51 (1): 60–68. https://doi.org/10.1002/ajim.20544.
Liu, C.-C., and J. J.-C. Ying. 2020. “DeepSafety: A deep learning framework for unsafe behaviors detection of steel activity in construction projects.” In Proc., 2020 Int. Computer Symp. (ICS), 135–140. New York: IEEE. https://doi.org/10.1109/ics51289.2020.00036.
Liu, M., D. Hong, S. Han, and S. Lee. 2016. “Silhouette-based on-site human action recognition in single-view video.” In Proc., Construction Research Congress 2016, 951–959. Reston, VA: ASCE. https://doi.org/10.1061/9780784479827.096.
Liu, T., Y. Song, Y. Gu, and A. Li. 2013. “Human action recognition based on depth images from microsoft kinect.” In Proc., 2013 4th Global Congress on Intelligent Systems, 200–204. New York: IEEE. https://doi.org/10.1109/gcis.2013.38.
Luo, H., M. Wang, P. K.-Y. Wong, and J. C. P. Cheng. 2020. “Full body pose estimation of construction equipment using computer vision and deep learning techniques.” Autom. Constr. 110 (Feb): 103016. https://doi.org/10.1016/j.autcon.2019.103016.
Ma, Y., and L. Zhu. 2013. “A review on dimension reduction.” Int. Stat. Rev. 81 (1): 134–150. https://doi.org/10.1111/j.1751-5823.2012.00182.x.
MOHURD (Ministry of Housing and Urban-Rural Development of the People’s Republic of China). 2019. “Notification of production safety accidents in housing and municipal engineering.” Accessed September 7, 2021. https://www.mohurd.gov.cn/gongkai/fdzdgknr/tzgg/202006/20200624_246031.html.
Nadhim, E. A., C. Hon, B. Xia, I. Stewart, and D. Fang. 2016. “Falls from height in the construction industry: A critical review of the scientific literature.” Int. J. Environ. Res. Public Health 13 (7): 638. https://doi.org/10.3390/ijerph13070638.
Naveenkumar, M., and S. Domnic. 2020. “Deep ensemble network using distance maps and body part features for skeleton based action recognition.” Pattern Recognit. 100 (Apr): 107125. https://doi.org/10.1016/j.patcog.2019.107125.
Núñez, J. C., R. Cabido, J. J. Pantrigo, A. S. Montemayor, and J. F. Vélez. 2018. “Convolutional neural networks and long short-term memory for skeleton-based human activity and hand gesture recognition.” Pattern Recognit. 76 (Apr): 80–94. https://doi.org/10.1016/j.patcog.2017.10.033.
OSHA (Occupational Safety and Health Administration). 2015. “Fall protection in construction.” Accessed September 20, 2022. https://www.osha.gov/sites/default/files/publications/OSHA3146.pdf.
Piyathilaka, L., and S. Kodagoda. 2013. “Gaussian mixture based HMM for human daily activity recognition using 3D skeleton features.” In Proc., 2013 IEEE 8th Conf. on Industrial Electronics and Applications (ICIEA), 567–572. New York: IEEE. https://doi.org/10.1109/ICIEA.2013.6566433.
Plagemann, C., V. Ganapathi, D. Koller, and S. Thrun. 2010. “Real-time identification and localization of body parts from depth images.” In Proc., 2010 IEEE Int. Conf. on Robotics and Automation, 3108–3113. New York: IEEE. https://doi.org/10.1109/ROBOT.2010.5509559.
Ray, S. J., and J. Teizer. 2012. “Real-time construction worker posture analysis for ergonomics training.” Adv. Eng. Inf. 26 (2): 439–455. https://doi.org/10.1016/j.aei.2012.02.011.
Rey-Merchan, M. D. C., J. M. Gomez-de-Gabriel, J. A. Fernandez-Madrigal, and A. Lopez-Arquillos. 2020. “Improving the prevention of fall from height on construction sites through the combination of technologies.” Int. J. Occup. Saf. Ergon. 28 (1): 590–599. https://doi.org/10.1080/10803548.2020.1815393.
Sa, J., D.-C. Seo, and S. D. Choi. 2009. “Comparison of risk factors for falls from height between commercial and residential roofers.” J. Saf. Res. 40 (1): 1–6. https://doi.org/10.1016/j.jsr.2008.10.010.
Schmidhuber, J. 2015. “Deep learning in neural networks: An overview.” Neural Networks 61 (Jan): 85–117. https://doi.org/10.1016/j.neunet.2014.09.003.
Seo, J., S. Han, S. Lee, and H. Kim. 2015. “Computer vision techniques for construction safety and health monitoring.” Adv. Eng. Inf. 29 (2): 239–251. https://doi.org/10.1016/j.aei.2015.02.001.
Serpen, G., and R. H. Khan. 2018. “Real-time detection of human falls in progress: Machine learning approach.” Procedia Comput. Sci. 140 (Jan): 238–247. https://doi.org/10.1016/j.procs.2018.10.324.
Shoaib, M., S. Bosch, O. D. Incel, H. Scholten, and P. J. M. Havinga. 2015. “A survey of online activity recognition using mobile phones.” Sensors 15 (1): 2059–2085. https://doi.org/10.3390/s150102059.
Simonyan, K., and A. Zisserman. 2014. “Two-stream convolutional networks for action recognition in videos.” Preprint, submitted June 9, 2014. https://arxiv.org/abs/1406.2199.
Tamura, T., T. Yoshimura, M. Sekine, M. Uchida, and O. Tanaka. 2009. “A wearable airbag to prevent fall injuries.” IEEE Trans. Inf. Technol. Biomed. 13 (6): 910–914. https://doi.org/10.1109/TITB.2009.2033673.
The Eastman Kodak Company. 2003. Kodak’s ergonomic design for people at work. New York: Wiley. https://doi.org/10.1002/9780470172469.
Wang, L., D. Q. Huynh, and P. Koniusz. 2020. “A comparative review of recent kinect-based action recognition algorithms.” IEEE Trans. Image Process. 29 (Jul): 15–28. https://doi.org/10.1109/TIP.2019.2925285.
Yan, X., H. Zhang, and H. Li. 2020. “Computer vision-based recognition of 3D relationship between construction entities for monitoring struck-by accidents.” Comput.-Aided Civ. Infrastruct. Eng. 35 (9): 1023–1038. https://doi.org/10.1111/mice.12536.
Yun, H., E. Nakamura, Y. Kageyama, C. Ishizawa, N. Kato, K. Igarashi, and M. Suzuki. 2021. “Development of action-recognition technology using LSTM based on skeleton data.” In Proc., 9th IIAE Int. Conf. on Industrial Application Engineering 2020, 35–39. Fukuoka, Japan: Institute of Industrial Applications Engineers. https://doi.org/10.12792/iciae2021.009.
Zhang, M., and D. Fang. 2013. “A cognitive analysis of why Chinese scaffolders do not use safety harnesses in construction.” Construct. Manage. Econ. 31 (3): 207–222. https://doi.org/10.1080/01446193.2013.764000.
Zhang, M., R. Shi, and Z. Yang. 2020. “A critical review of vision-based occupational health and safety monitoring of construction site workers.” Saf. Sci. 126 (Jun): 104658. https://doi.org/10.1016/j.ssci.2020.104658.
Zhang, P., N. Li, Z. Jiang, D. Fang, and C. J. Anumba. 2019. “An agent-based modeling approach for understanding the effect of worker-management interactions on construction workers’ safety-related behaviors.” Autom. Constr. 97 (Jan): 29–43. https://doi.org/10.1016/j.autcon.2018.10.015.
Zhang, S., Y. Yang, J. Xiao, X. Liu, Y. Yang, D. Xie, and Y. Zhuang. 2018. “Fusing geometric features for skeleton-based action recognition using multilayer LSTM networks.” IEEE Trans. Multimedia 20 (9): 2330–2343. https://doi.org/10.1109/TMM.2018.2802648.
Zhao, J., and E. Obonyo. 2020. “Convolutional long short-term memory model for recognizing construction workers’ postures from wearable inertial measurement units.” Adv. Eng. Inf. 46 (Oct): 101177. https://doi.org/10.1016/j.aei.2020.101177.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 149 • Issue 4 • April 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Apr 7, 2022
Accepted: Dec 12, 2022
Published online: Feb 13, 2023
Published in print: Apr 1, 2023
Discussion open until: Jul 13, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- He Huang, Hao Hu, Feng Xu, Zhipeng Zhang, Kinesiology-Inspired Assessment of Intrusion Risk Based on Human Motion Features, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-13638, 150, 7, (2024).
- Xiao Lin, Ziyang Guo, Hongling Guo, Ying Zhou, Vision-Based Real-Time Posture Tracking for Multiple Construction Workers, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5816, 38, 4, (2024).