Identifying Unsafe Behavior of Construction Workers: A Dynamic Approach Combining Skeleton Information and Spatiotemporal Features
Publication: Journal of Construction Engineering and Management
Volume 149, Issue 11
Abstract
Vision-based methods for action recognition are valuable for supervising construction workers’ unsafe behaviors. However, current monitoring methods have limitations in extracting dynamic information about workers. Identifying hazardous actions based on the spatiotemporal relationships between workers’ skeletal points remains a significant challenge in construction sites. This paper proposed an automated method for recognizing dynamic hazardous actions. The method used the OpenPose network to extract workers’ skeleton information from the video and applied a spatiotemporal graph convolutional network (ST-GCN) to analyze the dynamic spatiotemporal relationships between workers’ body skeletons, enabling automatic recognition of hazardous actions. A novel human partitioning strategy and nonlocal attention mechanism were designed to assign appropriate weight parameters to different joints involved in actions, thereby improving the recognition accuracy of complex construction actions. The enhanced model is called the attention module spatiotemporal graph convolutional network (AM-STGCN). The method achieved a test accuracy of 90.50% and 87.08% in typical work scenarios, namely high-altitude scaffolding scenes with close-up and far views, surpassing the performance of the original ST-GCN model. The high-accuracy test results demonstrate that the model can accurately identify workers’ hazardous actions. The newly proposed model is inferred to have promising application prospects and the potential to be applied in broader construction scenarios for on-site monitoring of hazardous actions.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This research is supported by the National Natural Science Foundation of China (Grant No. 72071097), MOE (Ministry of Education in China) Project of Humanities and Social Sciences (Grant No. 20YJAZH034), and the 16th Talent Summit Program of Six Major Fields in Jiangsu Province (Grant No. SZCY-014).
References
Akhavian, R., and A. H. Behzadan. 2016. “Smartphone-based construction workers’ activity recognition and classification.” Autom. Constr. 71 (Nov): 198–209. https://doi.org/10.1016/j.autcon.2016.08.015.
Bangaru, S. S., C. Wang, S. A. Busam, and F. Aghazadeh. 2021. “ANN-based automated scaffold builder activity recognition through wearable EMG and IMU sensors.” Autom. Constr. 126 (Jun): 103653. https://doi.org/10.1016/j.autcon.2021.103653.
Cheng, T., J. Teizer, G. C. Migliaccio, and U. C. Gatti. 2013. “Automated task-level activity analysis through fusion of real time location sensors and worker’s thoracic posture data.” Autom. Constr. 29 (Jan): 24–39. https://doi.org/10.1016/j.autcon.2012.08.003.
Ding, L., W. Fang, H. Luo, P. E. D. Love, B. Zhong, and X. Ouyang. 2018. “A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory.” Autom. Constr. 86 (Feb): 118–124. https://doi.org/10.1016/j.autcon.2017.11.002.
Escorcia, V., M. A. Dávila, M. Golparvar-Fard, and J. C. Niebles. 2012. “Automated vision-based recognition of construction worker actions for building interior construction operations using RGB-D cameras.” In Proc., Construction Research Congress 2012: Construction Challenges in a Flat World, 879–888. Reston, VA: ASCE.
Fang, W., L. Ding, P. E. D. Love, H. Luo, H. Li, and F. Peña-Mora. 2020. “Computer vision applications in construction safety assurance.” Autom. Constr. 110 (Feb): 103013.
Fang, W., L. Ding, H. Luo, and P. E. D. Love. 2018. “Falls from heights: A computer vision-based approach for safety harness detection.” Autom. Constr. 91 (Jul): 53–61. https://doi.org/10.1016/j.autcon.2018.02.018.
Fang, W., B. Zhong, N. Zhao, P. E. D. Love, H. Luo, and J. Xue. 2019. “A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network.” Adv. Eng. Inf. 39 (Jan): 170–177. https://doi.org/10.1016/j.aei.2018.12.005.
Fang, Y. C., and R. J. Dzeng. 2014. “A smartphone-based detection of fall portents for construction workers.” Procedia Eng. 85 (Jan): 147–156. https://doi.org/10.1016/j.proeng.2014.10.539.
Feng, Y., and Y. Li. 2018. “An overview of deep learning optimization methods and learning rate attenuation methods.” Hans J. Data Min. 8 (4): 186–200. https://doi.org/10.12677/hjdm.2018.84020.
Gong, Y., K. Yang, J. Seo, and J. G. Lee. 2022. “Wearable acceleration-based action recognition for long-term and continuous activity analysis in construction site.” J. Build. Eng. 52 (Jul): 104448. https://doi.org/10.1016/j.jobe.2022.104448.
Guo, G., and A. Lai. 2014. “A survey on still image based human action recognition.” Pattern Recognit. 47 (10): 3343–3361. https://doi.org/10.1016/j.patcog.2014.04.018.
Guo, H., Y. Yu, Q. Ding, and M. Skitmore. 2018. “Image-and-skeleton-based parameterized approach to real-time identification of construction workers’ unsafe behaviors.” J. Constr. Eng. Manage. 144 (6): 04018042. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001497.
Han, S., and S. Lee. 2013. “A vision-based motion capture and recognition framework for behavior-based safety management.” Autom. Constr. 35 (Nov): 131–141. https://doi.org/10.1016/j.autcon.2013.05.001.
Heinrich, H. W. 1941. Industrial accident prevention. A scientific approach. 2nd ed. New York: McGraw-Hill.
HSE (Health and Safety Executive). 2022. “Work-related fatal injuries in Great Britain.” Accessed August 3, 2022. https://www.hse.gov.uk/statistics/fatals.html.
Jebelli, H., C. R. Ahn, and T. L. Stentz. 2016. “Fall risk analysis of construction workers using inertial measurement units: Validating the usefulness of the postural stability metrics in construction.” Saf. Sci. 84 (Apr): 161–170. https://doi.org/10.1016/j.ssci.2015.12.012.
Jegham, I., A. Ben Khalifa, I. Alouani, and M. A. Mahjoub. 2020. “Vision-based human action recognition: An overview and real-world challenges.” Forensic Sci. Int.: Digital Invest. 32 (Mar): 200901. https://doi.org/10.1016/j.fsidi.2019.200901.
Kim, H., K. Kim, and H. Kim. 2016. “Vision-based object-centric safety assessment using fuzzy inference: Monitoring struck-by accidents with moving objects.” J. Comput. Civ. Eng. 30 (4): 04015075. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000562.
Kong, T., W. Fang, P. E. D. Love, H. Luo, S. Xu, and H. Li. 2021. “Computer vision and long short-term memory: Learning to predict unsafe behaviour in construction.” Adv. Eng. Inf. 50 (Oct): 101400. https://doi.org/10.1016/j.aei.2021.101400.
Kong, Y., L. Li, and K. Zhang. 2019. “Attention module-based spatial–temporal graph convolutional networks for skeleton-based action recognition.” J. Electron. Imaging 28 (4): 1. https://doi.org/10.1117/1.JEI.28.4.043032.
Lan, Z., Y. Zhu, A. G. Hauptmann, and S. Newsam. 2017. “Deep local video feature for action recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition Workshops, 1–7. New York: IEEE.
Laptev, I. 2005. “On space-time interest points.” Int. J. Comput. Vis. 64 (2): 107–123. https://doi.org/10.1007/s11263-005-1838-7.
Li, J., B. Li, J. Xu, R. Xiong, and W. Gao. 2018. “Fully connected network-based intra prediction for image coding.” IEEE Trans. Image Process. 27 (7): 3236–3247. https://doi.org/10.1109/TIP.2018.2817044.
Li, Z., and D. Li. 2022. “Action recognition of construction workers under occlusion.” J. Build. Eng. 45 (Jan): 103352. https://doi.org/10.1016/j.jobe.2021.103352.
Memarzadeh, M., M. Golparvar-Fard, and J. C. Niebles. 2013. “Automated 2D detection of construction equipment and workers from site video streams using histograms of oriented gradients and colors.” Autom. Constr. 32 (Jul): 24–37. https://doi.org/10.1016/j.autcon.2012.12.002.
MOHURD (Ministry of Housing and Urban-Rural Development). 2022. “Circular on the production safety accidents of housing and municipal engineering in 2020.” Accessed June 19, 2022. https://www.mohurd.Gov.cn/gongkaifdzdgknr/tzgg/202006/20200624_246031.html.
Ogunseiju, O. R., J. Olayiwola, A. A. Akanmu, and N. Chukwuma. 2022. “Recognition of workers’ actions from time-series signal images using deep convolutional neural network.” Smart Sustainable Built Environ. 11 (4): 812–831. https://doi.org/10.1108/SASBE-11-2020-0170.
Park, M. W., N. Elsafty, and Z. Zhu. 2015. “Hardhat-wearing detection for enhancing on-site safety of construction workers.” J. Constr. Eng. Manage. 141 (9): 04015024. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000974.
Pinto, A., I. L. Nunes, and R. A. Ribeiro. 2011. “Occupational risk assessment in construction industry-overview and reflection.” Saf. Sci. 49 (5): 616–624. https://doi.org/10.1016/j.ssci.2011.01.003.
Sahoo, S. P., S. Modalavalasa, and S. Ari. 2022. “DISNet: A sequential learning framework to handle occlusion in human action recognition with video acquisition sensors.” Digital Signal Process. 131 (Nov): 103763. https://doi.org/10.1016/j.dsp.2022.103763.
Si, C., W. Chen, W. Wang, and T. Tan. 2019. “An attention enhanced graph convolutional LSTM network for skeleton-based action recognition.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 1227–1236. New York: IEEE.
Simonyan, K., and A. Zisserman. 2015. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. https://arxiv.org/abs/1409.1556.
Tang, S., M. Golparvar-Fard, M. Naphade, and M. Gopalakrishna. 2020. “Video-based motion trajectory forecasting method for proactive construction safety monitoring systems.” J. Comput. Civ. Eng. 34 (6): 04020041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000923.
Trabelsi, R., I. Jabri, F. Melgani, F. Smach, N. Conci, and A. Bouallegue. 2019. “Indoor object recognition in RGBD images with complex-valued neural networks for visually-impaired people.” Neurocomputing 330 (Feb): 94–103. https://doi.org/10.1016/j.neucom.2018.11.032.
Tran, D., L. Bourdev, R. Fergus, L. Torresani, and M. Paluri. 2015. “Learning spatiotemporal features with 3D convolutional networks.” In Proc., IEEE Int. Conf. on Computer Vision, 4489–4497. New York: IEEE.
Vasconez, J. P., H. Admoni, and F. Auat-Cheein. 2021. “A methodology for semantic action recognition based on pose and human-object interaction in avocado harvesting processes.” Comput. Electron. Agric. 184 (May): 106057. https://doi.org/10.1016/j.compag.2021.106057.
Wang, X., R. Girshick, A. Gupta, and K. He. 2018. “Non-local neural networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 7794–7803. New York: IEEE.
Wei, S. E., V. Ramakrishna, T. Kanade, and Y. Sheikh. 2016. “Convolutional pose machines.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 4724–4732. New York: IEEE.
Weiss, K., T. M. Khoshgoftaar, and D. Wang. 2016. “A survey of transfer learning.” J. Big Data 3 (1): 1–40. https://doi.org/10.1186/s40537-016-0043-6.
Wen, J., Y. Shen, and J. Yang. 2022. “Multi-view gait recognition based on generative adversarial network.” Neural Process. Lett. 54 (3): 1855–1877. https://doi.org/10.1007/s11063-021-10709-1.
Wu, Z., X. Wang, Y. G. Jiang, H. Ye, and X. Xue. 2015. “Modeling spatial-temporal clues in a hybrid deep learning framework for video classification.” In Proc., 23rd ACM Int. Conf. on Multimedia, 461–470. New York: Association for Computing Machinery.
Yan, S., Y. Xiong, and D. Lin. 2018. “Spatial temporal graph convolutional networks for skeleton-based action recognition.” In Proc., Thirty-Second AAAI Conf. on Artificial Intelligence. Palo Alto, CA: AAAI.
Yang, J. 2018. “Enhancing action recognition of construction workers using data-driven scene parsing.” J. Civ. Eng. Manage. 24 (7): 568–580. https://doi.org/10.3846/jcem.2018.6133.
Yang, J., Z. Shi, and Z. Wu. 2016. “Vision-based action recognition of construction workers using dense trajectories.” Adv. Eng. Inf. 30 (3): 327–336. https://doi.org/10.1016/j.aei.2016.04.009.
Yu, Y., H. Guo, Q. Ding, H. Li, and M. Skitmore. 2017. “An experimental study of real-time identification of construction workers’ unsafe behaviors.” Autom. Constr. 82 (Oct): 193–206. https://doi.org/10.1016/j.autcon.2017.05.002.
Zhang, P., C. Lan, J. Xing, W. Zeng, J. Xue, and N. Zheng. 2019. “View adaptive neural networks for high performance skeleton-based human action recognition.” IEEE Trans. Pattern Anal. Mach. Intell. 41 (8): 1963–1978. https://doi.org/10.1109/TPAMI.2019.2896631.
Zhao, R., W. Xu, H. Su, and Q. Ji. 2019. “Bayesian hierarchical dynamic model for human action recognition.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 7733–7742. New York: IEEE.
Zhou, B., A. Andonian, A. Oliva, and A. Torralba. 2018. “Temporal relational reasoning in videos.” In Proc., European Conf. on Computer Vision, 803–818. Berlin: Springer.
Zhou, K., T. Wu, C. Wang, J. Wang, and C. Li. 2020. “Skeleton based abnormal behavior recognition using spatio-temporal convolution and attention-based LSTM.” Procedia Comput. Sci. 174 (Jan): 424–432. https://doi.org/10.1016/j.procs.2020.06.110.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 149 • Issue 11 • November 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Feb 2, 2023
Accepted: Jul 20, 2023
Published online: Sep 11, 2023
Published in print: Nov 1, 2023
Discussion open until: Feb 11, 2024
ASCE Technical Topics:
- Business management
- Construction engineering
- Construction management
- Construction methods
- Disaster risk management
- Disasters and hazards
- Employment
- Engineering fundamentals
- Human and behavioral factors
- Labor
- Model accuracy
- Models (by type)
- Occupational safety
- Personnel management
- Practice and Profession
- Public administration
- Public health and safety
- Risk management
- Safety
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.