Vision-Based Detection of Unsafe Worker Guardrail Climbing Based on Posture and Instance Segmentation Data Fusion
Publication: Journal of Construction Engineering and Management
Volume 150, Issue 11
Abstract
Currently, the incidence of accidents involving falls from height at construction sites caused by workers climbing guardrails is still high. Traditional unsafe behavior management mainly relies on a safety patrol of construction-site supervisors, which consumes considerable laborpower and time. There is still a critical need for an automated safety management method to identify unsafe guardrail climbing behavior. This study proposes a worker behavior identification method based on visual data fusion of a worker’s surrounding environment and posture data. Videos of seven participants’ guardrail climbing behavior through multiangle and multidistance cameras were analyzed to verify this method. By analyzing the environment and posture of the participants, three methods based on environment, posture, and fusion data were used to detect the stage of guardrail climbing action of the workers and compare them with the ground truth labeled by safety experts. The precision and recall of worker guardrail climbing behavior based on the fusion method were 82% and 83% respectively, which is better performance than that obtained using a single method. The data fusion–based method avoids the misjudgment generated by a single detection method and can identify the guardrail climbing behavior more accurately.
Practical Applications
Guardrail climbing is a typical unsafe behavior that exposes workers to a high risk of falling from height. However, there is a lack of research on the interaction between workers and guardrail systems in the construction industry. This study provides a nonintrusive method for automating detection and management of guardrail climbing behavior on construction site. Using existing surveillance cameras, this method can be deployed at low cost with slight interference with workers. Based on the detection, appropriate interventions are expected to effectively reduce workers’ unsafe behaviors during construction and improve safety on site. The detection of guardrail climbing, which is one of the variety of unsafe behaviors associated with falls from height, can enrich the intelligent construction safety management system effectively. Moreover, this study also provides reference and quantitative indicators (e.g., a guardrail climbing unsafe behavior database) for risk assessment and early warning of workers who are exposed to risk of fall from height.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This research was supported by the Science Research Plan of Shanghai Municipal Science and Technology Committee (Grant No. 20dz1201301), and the 2021 Science Research Plan of Shanghai Housing and Urban-Rural Development Management Committee (Grant No. 2021-002-4049).
References
Antwi-Afari, M. F., and H. Li. 2018. “Fall risk assessment of construction workers based on biomechanical gait stability parameters using wearable insole pressure system.” Adv. Eng. Inf. 38 (Oct): 683–694. https://doi.org/10.1016/j.aei.2018.10.002.
Baduge, S. K., S. Thilakarathna, J. S. Perera, M. Arashpour, P. Sharafi, B. Teodosio, A. Shringi, and P. Mendis. 2022. “Artificial intelligence and smart vision for building and construction 4.0: Machine and deep learning methods and applications.” Autom. Constr. 141 (Sep): 104440. https://doi.org/10.1016/j.autcon.2022.104440.
Bobick, T. G., E. A. McKenzie Jr., and T.-Y. Kau. 2010. “Evaluation of guardrail systems for preventing falls through roof and floor holes.” J. Saf. Res. 41 (3): 203–211. https://doi.org/10.1016/j.jsr.2010.02.008.
Bolya, D., Z. Chong, X. Fanyi, and Y. J. Lee. 2019. “YOLACT++: Better real-time instance segmentation.” Preprint, submitted December 3, 2019. https://arxiv.org/abs/1912.06218.
Cai, Z., and N. Vasconcelos. 2021. “Cascade R-CNN: High quality object detection and instance segmentation.” IEEE Trans. Pattern Anal. Mach. Intell. 43 (5): 1483–1498. https://doi.org/10.1109/TPAMI.2019.2956516.
Chen, H., X. Luo, Z. Zheng, and J. Ke. 2019. “A proactive workers’ safety risk evaluation framework based on position and posture data fusion.” Autom. Constr. 98 (Feb): 275–288. https://doi.org/10.1016/j.autcon.2018.11.026.
Chen, H., X. Luo, and Z. Zhu. 2021. “A worker posture coding scheme to link automatic and manual coding.” Autom. Constr. 125 (May): 103630. https://doi.org/10.1016/j.autcon.2021.103630.
Cheng, T., G. C. Migliaccio, J. Teizer, and U. C. Gatti. 2013. “Data fusion of real-time location sensing and physiological status monitoring for ergonomics analysis of construction workers.” J. Comput. Civ. Eng. 27 (3): 320–335. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000222.
Chi, C.-F., T.-C. Chang, and H.-I. Ting. 2005. “Accident patterns and prevention measures for fatal occupational falls in the construction industry.” Appl. Ergon. 36 (4): 391–400. https://doi.org/10.1016/j.apergo.2004.09.011.
Chian, E., W. Fang, Y. M. Goh, and J. Tian. 2021. “Computer vision approaches for detecting missing barricades.” Autom. Constr. 131 (Nov): 103862. https://doi.org/10.1016/j.autcon.2021.103862.
Choo, H., B. Lee, H. Kim, and B. Choi. 2023. “Automated detection of construction work at heights and deployment of safety hooks using IMU with a barometer.” Autom. Constr. 147 (Mar): 104714. https://doi.org/10.1016/j.autcon.2022.104714.
Ding, L., W. Fang, H. Luo, P. E. D. Love, B. Zhong, and X. Ouyang. 2018. “A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory.” Autom. Constr. 86 (Feb): 118–124. https://doi.org/10.1016/j.autcon.2017.11.002.
Fang, Q., H. Li, X. Luo, L. Ding, H. Luo, and C. Li. 2018. “Computer vision aided inspection on falling prevention measures for steeplejacks in an aerial environment.” Autom. Constr. 93 (Sep): 148–164. https://doi.org/10.1016/j.autcon.2018.05.022.
Fang, W., L. Ding, P. E. D. Love, H. Luo, H. Li, F. Peña-Mora, B. Zhong, and C. Zhou. 2020a. “Computer vision applications in construction safety assurance.” Autom. Constr. 110 (Feb): 103013. https://doi.org/10.1016/j.autcon.2019.103013.
Fang, W., P. E. D. Love, H. Luo, and L. Ding. 2020b. “Computer vision for behaviour-based safety in construction: A review and future directions.” Adv. Eng. Inf. 43 (Jan): 100980. https://doi.org/10.1016/j.aei.2019.100980.
Fang, W., P. E. D. Love, H. Luo, and S. Xu. 2022. “A deep learning fusion approach to retrieve images of people’s unsafe behavior from construction sites.” Dev. Built Environ. 12 (Dec): 100085. https://doi.org/10.1016/j.dibe.2022.100085.
Fang, W., B. Zhong, N. Zhao, P. E. D. Love, H. Luo, J. Xue, and S. Xu. 2019. “A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network.” Adv. Eng. Inf. 39 (Jan): 170–177. https://doi.org/10.1016/j.aei.2018.12.005.
Galy, B. 2021. “In situ strength testing of temporary steel guardrails installed on existing flat-roof parapets.” Pract. Period. Struct. Des. Constr. 26 (1): 04020047. https://doi.org/10.1061/(ASCE)SC.1943-5576.0000535.
Galy, B., and A. Lan. 2020. “Strength of temporary wooden guardrails installed on open joists and prefabricated walls.” Pract. Period. Struct. Des. Constr. 25 (2): 04020006. https://doi.org/10.1061/(ASCE)SC.1943-5576.0000473.
Goh, Y. M., and N. F. B. Sa’adon. 2015. “Cognitive factors influencing safety behavior at height: A multimethod exploratory study.” J. Constr. Eng. Manage. 141 (6): 04015003. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000972.
Guo, H., W. Liu, W. Zhang, and M. Skitmor. 2014. “A BIM-RFID unsafe on-site behavior warning system.” In Proc., ICCREM 2014: Smart Construction and Management in the Context of New Technology, 330–339. Reston, VA: ASCE.
Guo, H., Y. Yu, Q. Ding, and M. Skitmore. 2018. “Image-and-skeleton-based parameterized approach to real-time identification of construction workers’ unsafe behaviors.” J. Constr. Eng. Manage. 144 (6): 04018042. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001497.
Halabi, Y., H. Xu, D. B. Long, Y. H. Chen, Z. X. Yu, F. Alhaek, and W. Alhaddad. 2022. “Causal factors and risk assessment of fall accidents in the US construction industry: A comprehensive data analysis (2000–2020).” Saf. Sci. 146 (Feb): 105537. https://doi.org/10.1016/j.ssci.2021.105537.
Han, S., S. Lee, and F. Peña-Mora. 2013. “Vision-based detection of unsafe actions of a construction worker: Case study of ladder climbing.” J. Comput. Civ. Eng. 27 (6): 635–644. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000279.
Haslam, R. A., S. A. Hide, A. G. F. Gibb, D. E. Gyi, T. Pavitt, S. Atkinson, and A. R. Duff. 2005. “Contributing factors in construction accidents.” Appl. Ergon. 36 (4): 401–415. https://doi.org/10.1016/j.apergo.2004.12.002.
He, C., G. Jia, H. Zhang, and K. Ding. 2020. “Spatiotemporal characteristics and behavioral factors of fatal construction accidents in China, 2012–2018.” In Proc., Construction Research Congress 2020, 195–203. Reston, VA: ASCE.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask R-CNN.” In Proc., 2017 IEEE Int. Conf. on Computer Vision (ICCV), 2980–2988. New York: IEEE.
Huang, H., H. Hu, F. Xu, Z. Zhang, and Y. Tao. 2023. “Skeleton-based automatic assessment and prediction of intrusion risk in construction hazardous areas.” Saf. Sci. 164 (Aug): 106150. https://doi.org/10.1016/j.ssci.2023.106150.
Ionescu, C., D. Papava, V. Olaru, and C. Sminchisescu. 2014. “Human3.6M: Large scale datasets and predictive methods for 3D human sensing in natural environments.” IEEE Trans. Pattern Anal. Mach. Intell. 36 (7): 1325–1339. https://doi.org/10.1109/TPAMI.2013.248.
Kang, Y. 2018. “Use of fall protection in the US construction industry.” J. Manage. Eng. 34 (6): 04018045. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000655.
Khan, M., R. Khalid, S. Anjum, S. V.-T. Tran, and C. Park. 2022. “Fall prevention from scaffolding using computer vision and IoT-based monitoring.” J. Constr. Eng. Manage. 148 (7): 04022051. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002278.
Khan, M., C. Nnaji, M. S. Khan, A. Ibrahim, D. Lee, and C. Park. 2023. “Risk factors and emerging technologies for preventing falls from heights at construction sites.” Autom. Constr. 153 (Sep): 104955. https://doi.org/10.1016/j.autcon.2023.104955.
Kirillov, A., W. Yuxin, H. Kaiming, and R. Girshick. 2020. “PointRend: Image segmentation as rendering.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition 2020, 9799–9808. New York: IEEE.
Kleiner, B. M., T. L. Smith-Jackson, T. H. Mills III, M. O’Brien, and E. Haro. 2008. “Design, development, and deployment of a rapid universal safety and health system for construction.” J. Constr. Eng. Manage. 134 (4): 273–279. https://doi.org/10.1061/(ASCE)0733-9364(2008)134:4(273).
Kolar, Z., H. Chen, and X. Luo. 2018. “Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images.” Autom. Constr. 89 (May): 58–70. https://doi.org/10.1016/j.autcon.2018.01.003.
Kolekar, M. H., N. Bharti, and P. N. Patil. 2016. “Detection of fence climbing using activity recognition by support vector machine classifier.” In Proc., IEEE Region 10 Conf. (TENCON), 398–402. New York: IEEE.
Lestari, R. I., B. H. W. Guo, and Y. M. Goh. 2019. “Causes, solutions, and adoption barriers of falls from roofs in the Singapore construction industry.” J. Constr. Eng. Manage. 145 (5): 04019027. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001649.
Li, P., Q. Wang, Z. Guo, T. Mei, Q. Li, S. Qiao, and W. Zuo. 2022. “Identifying falling-from-height hazards in building information models: A voxelization-based method.” J. Constr. Eng. Manage. 148 (2): 04021203. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002236.
Liu, J., H. Luo, and H. Liu. 2022. “Deep learning-based data analytics for safety in construction.” Autom. Constr. 140 (Aug): 104302. https://doi.org/10.1016/j.autcon.2022.104302.
Liu, M., S. Han, and S. Lee. 2017. “Potential of convolutional neural network-based 2D human pose estimation for on-site activity analysis of construction workers.” In Proc., Computing in Civil Engineering 2017, 141–149. New York: IEEE.
Mei, X., X. Zhou, F. Xu, and Z. Zhang. 2023. “Human intrusion detection in static hazardous areas at construction sites: Deep learning-based method.” J. Constr. Eng. Manage. 149 (1): 04022142. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002409.
NIOSH (National Institute for Occupational Safety and Health). 2015. “Hierarchy of controls.” Accessed November 24, 2023. https://www.cdc.gov/niosh/hierarchy-of-controls/about/?CDC_AAref_Val=https://www.cdc.gov/niosh/topics/hierarchy/.
Pavllo, D., C. Feichtenhofer, D. Grangier, and M. Auli. 2019. “3D human pose estimation in video with temporal convolutions and semi-supervised training.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition 2019, 7753–7762. New York: IEEE.
Rafindadi, A. D., M. Napiah, I. Othman, M. Mikić, A. Haruna, H. Alarifi, and Y. Y. Al-Ashmori. 2022. “Analysis of the causes and preventive measures of fatal fall-related accidents in the construction industry.” Ain Shams Eng. J. 13 (4): 101712. https://doi.org/10.1016/j.asej.2022.101712.
Rao, A. S., M. Radanovic, Y. Liu, S. Hu, Y. Fang, K. Khoshelham, M. Palaniswami, and T. Ngo. 2022. “Real-time monitoring of construction sites: Sensors, methods, and applications.” Autom. Constr. 136 (Apr): 104099. https://doi.org/10.1016/j.autcon.2021.104099.
Seo, J., S. Han, S. Lee, and H. Kim. 2015. “Computer vision techniques for construction safety and health monitoring.” Adv. Eng. Inf. 29 (2): 239–251. https://doi.org/10.1016/j.aei.2015.02.001.
Shahandashti, S. M., et al. 2011. “Data-fusion approaches and applications for construction engineering.” J. Constr. Eng. Manage. 137 (10): 863–869. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000287.
Tam, C. M., S. X. Zeng, and Z. M. Deng. 2004. “Identifying elements of poor construction safety management in China.” Saf. Sci. 42 (7): 569–586. https://doi.org/10.1016/j.ssci.2003.09.001.
Tanvi Newaz, M., M. Ershadi, L. Carothers, M. Jefferies, and P. Davis. 2022. “A review and assessment of technologies for addressing the risk of falling from height on construction sites.” Saf. Sci. 147 (Mar): 105618. https://doi.org/10.1016/j.ssci.2021.105618.
Thi-Ngoc-Diem, P., C. Si-Quych-Di, D. Quang-Thien, T. Hoang-Le-Chi, N. Thanh-Hai, and T. Thanh-Dien. 2023. “Detection of fence climbing behavior in surveillance videos using YOLO V4.” In Proc., Int. Conf. on Intelligent Systems & Networks, 418–425. Singapore: Springer.
Wang, T., K. Wang, J. Li, H. Yu, J. Bian, X. Zhao, and W. Shuai. 2017. “Fast recognition of human climbing fences in transformer substations.” In Proc., 9th Int. Conf. on Advanced Computational Intelligence (ICACI), 195–200. New York: IEEE.
Wang, X., R. Zhang, T. Kong, L. Li, and C. Shen. 2020. “SOLOv2: Dynamic and fast instance segmentation.” In Vol. 33 of Advances in neural information processing systems, 17721–17732. Cambridge, MA: MIT Press.
Wu, H., and J. Zhao. 2018. “An intelligent vision-based approach for helmet identification for work safety.” Comput. Ind. 100 (Sep): 267–277. https://doi.org/10.1016/j.compind.2018.03.037.
Wu, Y., and J. Cai. 2017. “Research on new-type smart city in China based on smart construction theory.” In Proc., ICCREM 2017, 347–354. Reston, VA: ASCE.
Xiang, Q., G. Ye, Y. Liu, Y. M. Goh, D. Wang, and T. He. 2023. “Cognitive mechanism of construction workers’ unsafe behavior: A systematic review.” Saf. Sci. 159 (Mar): 106037. https://doi.org/10.1016/j.ssci.2022.106037.
Xiao, B., and S.-C. Kang. 2021. “Development of an image data set of construction machines for deep learning object detection.” J. Comput. Civ. Eng. 35 (2): 05020005. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000945.
Xiao, B., H. Xiao, J. Wang, and Y. Chen. 2022. “Vision-based method for tracking workers by integrating deep learning instance segmentation in off-site construction.” Autom. Constr. 136 (Apr): 104148. https://doi.org/10.1016/j.autcon.2022.104148.
Yu, E., and J. K. Aggarwal. 2009. “Recognizing persons climbing fences.” Int. J. Pattern Recognit. Artif. Intell. 23 (7): 1309–1332. https://doi.org/10.1142/S0218001409007570.
Yu, J., Y. Jiang, Z. Wang, Z. Cao, and T. Huang. 2016. “UnitBox: An advanced object detection network.” In Proc., 24th ACM Int. Conf. on Multimedia, 516–520. Chesterfield, MO: Tetrasoft.
Yu, Y., H. Guo, Q. Ding, H. Li, and M. Skitmore. 2017. “An experimental study of real-time identification of construction workers’ unsafe behaviors.” Autom. Constr. 82 (Oct): 193–206. https://doi.org/10.1016/j.autcon.2017.05.002.
Zhang, M., Z. Cao, Z. Yang, and X. Zhao. 2020. “Utilizing computer vision and fuzzy inference to evaluate level of collision safety for workers and equipment in a dynamic environment.” J. Constr. Eng. Manage. 146 (6): 04020051. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001802.
Zhi, T., S. Chunhua, and C. Hao. 2020. “Conditional convolutions for instance segmentation.” In Proc., Computer Vision–ECCV 2020: 16th European Conf., 282–298. Singapore: Springer.
Zuluaga, C. M., A. Albert, and M. A. Winkel. 2020. “Improving safety, efficiency, and productivity: Evaluation of fall protection systems for bridge work using wearable technology and utility analysis.” J. Constr. Eng. Manage. 146 (2): 04019107. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001764.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 150 • Issue 11 • November 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Jul 14, 2023
Accepted: Jun 6, 2024
Published online: Aug 26, 2024
Published in print: Nov 1, 2024
Discussion open until: Jan 26, 2025
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.