Machine Learning-Based Risk Analysis for Construction Worker Safety from Ubiquitous Site Photos and Videos
Publication: Journal of Computing in Civil Engineering
Volume 35, Issue 6
Abstract
This paper proposes a new method for single-worker severity level prediction from already collected site images and video clips. Onsite safety observers often assess workers’ severity levels during construction activities. While risk analysis is key to improving long–term construction site safety, omnipresent monitoring is still time-consuming and costly to implement. The recent growth of visual data captured actively on construction sites has opened a new opportunity to increase the frequency of worker safety monitoring. This paper shows that a comprehensive vision-based assessment is the most informative to automatically infer worker severity level from images. Efficient computer vision models are presented to conduct this risk analysis. The method is validated on a challenging image dataset first of its kind. Specifically, the proposed method detects and evaluates the worker state from visual data, defined by (1) worker body posture, (2) the usage of personal protective equipment, (3) worker interactions with tools and materials, (4) the construction activity being performed, and (5) the presence of surrounding workplace hazards. To estimate the worker state, a multitasked recognition model is introduced that recognizes objects, activity, and keypoints from visual data simultaneously, taking 36.6% less time and 40.1% less memory while keeping comparably performances compared to a system running individual models for each subtask. Worker activity recognition is further improved with a spatio-temporal graph neural network model using recognized per-frame worker activity, detected bounding boxes of tools and materials, and estimated worker poses. Finally, severity levels are predicted by a trained classifier on a dataset of images of construction workers accompanied with ground truth severity level annotations. In the test dataset assembled from real-world projects, the severity level prediction model achieves 85.7% cross-validation accuracy in a bricklaying task and 86.6% cross-validation accuracy for a plastering task, demonstrating the potential for near real-time worker safety detection and severity assessment.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code used during the study were provided by a third party: All videos of construction workers used in this study were from Professor Jun Yang (Yang et al. 2016). Direct requests for these materials may be made to the provider as indicated in the “Acknowledgments.”
Acknowledgments
We would like to thank RAAMAC lab students for their suggestions and support of the present study. We thank Professor Jun Yang for granting access to her construction worker activity videos dataset. This material is based in part upon work supported by the National Science Foundation (NSF) under Grants CMMI 1446765 and CMMI 1544999. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the NSF.
References
Årnes, A., K. Sallhammar, K. Haslum, T. Brekne, M. E. G. Moe, and S. J. Knapskog. 2005. “Real-time risk assessment with network sensors and intrusion detection systems.” In Computational intelligence and security, edited by Y. Hao, J. Liu, Y.-P. Wang, Y.-M. Cheung, H. Yin, L. Jiao, J. Ma, and Y.-C. Jiao, 388–397. Berlin: Springer.
Azar, E. R., and B. McCabe. 2012. “Automated visual recognition of dump trucks in construction videos.” J. Comput. Civ. Eng. 26 (6): 769–781. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000179.
BLS (Bureau of Labor Statistics). 2020. “Injuries, illnesses, and fatalities.” Accessed October 1, 2020. https://www.bls.gov/iif/.
Buchholz, B., V. Paquet, L. Punnett, D. Lee, and S. Moir. 1996. “Path: A work sampling-based approach to ergonomic job analysis for construction and other non-repetitive work.” Appl. Ergon. 27 (3): 177–187. https://doi.org/10.1016/0003-6870(95)00078-X.
Cai, J., Y. Zhang, and H. Cai. 2019. “Two-step long short-term memory method for identifying construction activities through positional and attentional cues.” Autom. Constr. 106 (Oct): 102886. https://doi.org/10.1016/j.autcon.2019.102886.
Cheng, T., and J. Teizer. 2013. “Real-time resource location data collection and visualization technology for construction safety and activity monitoring applications.” Autom. Constr. 34 (Sep): 3–15. https://doi.org/10.1016/j.autcon.2012.10.017.
Choudhry, R. M. 2014. “Behavior-based safety on construction sites: A case study.” Accid. Anal. Prev. 70 (Sep): 14–23. https://doi.org/10.1016/j.aap.2014.03.007.
Chu, W., S. Han, X. Luo, and Z. Zhu. 2020. “Monocular vision-based framework for biomechanical analysis or ergonomic posture assessment in modular construction.” J. Comput. Civ. Eng. 34 (4): 04020018. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000897.
Chung, J., C. Gulcehre, K. Cho, and Y. Bengio. 2014. “Empirical evaluation of gated recurrent neural networks on sequence modeling.” Preprint, submitted December 11, 2014. https://arxiv.org/abs/1412.3555.
Ding, L., W. Fang, H. Luo, P. E. Love, B. Zhong, and X. Ouyang. 2018. “A deep hybrid learning model to detect unsafe behavior: Integrating convolution neural networks and long short-term memory.” Autom. Constr. 86 (Feb): 118–124. https://doi.org/10.1016/j.autcon.2017.11.002.
Fang, Q., H. Li, X. Luo, L. Ding, H. Luo, and C. Li. 2018a. “Computer vision aided inspection on falling prevention measures for steeplejacks in an aerial environment.” Autom. Constr. 93 (Sep): 148–164. https://doi.org/10.1016/j.autcon.2018.05.022.
Fang, Q., H. Li, X. Luo, L. Ding, H. Luo, T. M. Rose, and W. An. 2018b. “Detecting non-hardhat-use by a deep learning method from far-field surveillance videos.” Autom. Constr. 85 (Jan): 1–9. https://doi.org/10.1016/j.autcon.2017.09.018.
Fang, W., L. Ding, P. E. Love, H. Luo, H. Li, F. Peña-Mora, B. Zhong, and C. Zhou. 2020. “Computer vision applications in construction safety assurance.” Autom. Constr. 110 (Feb): 103013. https://doi.org/10.1016/j.autcon.2019.103013.
Fang, W., L. Ding, H. Luo, and P. E. Love. 2018c. “Falls from heights: A computer vision-based approach for safety harness detection.” Autom. Constr. 91 (Jul): 53–61. https://doi.org/10.1016/j.autcon.2018.02.018.
Fleiss, J. L., and J. Cohen. 1973. “The equivalence of weighted kappa and the intraclass correlation coefficient as measures of reliability.” Educ. Psychol. Meas. 33 (3): 613–619. https://doi.org/10.1177/001316447303300309.
Golparvar-Fard, M., A. Heydarian, and J. C. Niebles. 2013. “Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers.” Adv. Eng. Inf. 27 (4): 652–663. https://doi.org/10.1016/j.aei.2013.09.001.
Hallowell, M. R., and J. A. Gambatese. 2010. “Population and initial validation of a formal model for construction safety risk management.” J. Constr. Eng. Manage. 136 (9): 981–990. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000204.
Han, K. K., and M. Golparvar-Fard. 2017. “Potential of big visual data and building information modeling for construction performance analytics: An exploratory study.” Autom. Constr. 73 (Jan): 184–198. https://doi.org/10.1016/j.autcon.2016.11.004.
Han, S., S. Lee, and F. Peña-Mora. 2013. “Vision-based detection of unsafe actions of a construction worker: Case study of ladder climbing.” J. Comput. Civ. Eng. 27 (6): 635–644. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000279.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask r-CNN.” In Vol. 2980–2988 of Proc., 2017 IEEE Int. Conf. on Computer Vision. New York: IEEE.
Heinrich, H. W. 1941. “Industrial accident prevention: A scientific approach.” In Industrial accident prevention: A scientific approach. New York: McGraw-Hill Book Company.
Hignett, S., and L. McAtamney. 2000. “Rapid entire body assessment (REBA).” Appl. Ergon. 31 (2): 201–205. https://doi.org/10.1016/S0003-6870(99)00039-3.
Jain, A., A. R. Zamir, S. Savarese, and A. Saxena. 2016. “Structural-RNN: Deep learning on spatio-temporal graphs.” In Proc., 2016 IEEE Conf. on Computer Vision and Pattern Recognition, 5308–5317. New York: IEEE.
Karhu, O., P. Kansi, and I. Kuorinka. 1977. “Correcting working postures in industry: A practical method for analysis.” Appl. Ergon. 8 (4): 199–201. https://doi.org/10.1016/0003-6870(77)90164-8.
Khosrowpour, A., J. C. Niebles, and M. Golparvar-Fard. 2014. “Vision-based workface assessment using depth images for activity analysis of interior construction operations.” Autom. Constr. 48 (Dec): 74–87. https://doi.org/10.1016/j.autcon.2014.08.003.
Lin, J. J., K. K. Han, and M. Golparvar-Fard. 2015. A framework for model-driven acquisition and analytics of visual data using UAVs for automated construction progress monitoring, 156–164. Reston, VA: ASCE. https://doi.org/10.1061/9780784479247.020.
Lin, T.-Y., P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. 2017. “Feature pyramid networks for object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 2117–2125. New York: IEEE.
Lin, T.-Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. 2014. “Microsoft coco: Common objects in context.” In Proc., Computer Vision—European Conf. on Computer Vision 2014, edited by D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars, 740–755. Cham, Switzerland: Springer.
Liu, M., S. Han, and S. Lee. 2016. “Tracking-based 3d human skeleton extraction from stereo video camera toward an on-site safety and ergonomic analysis.” Constr. Innovation 16 (3): 348–367. https://doi.org/10.1108/CI-10-2015-0054.
Luo, H., C. Xiong, W. Fang, P. E. Love, B. Zhang, and X. Ouyang. 2018a. “Convolutional neural networks: Computer vision-based workforce activity assessment in construction.” Autom. Constr. 94 (Oct): 282–289. https://doi.org/10.1016/j.autcon.2018.06.007.
Luo, X., H. Li, D. Cao, Y. Yu, X. Yang, and T. Huang. 2018b. “Towards efficient and objective work sampling: Recognizing workers’ activities in site surveillance videos with two-stream convolutional networks.” Autom. Constr. 94 (Oct): 360–370. https://doi.org/10.1016/j.autcon.2018.07.011.
Luo, X., H. Li, H. Wang, Z. Wu, F. Dai, and D. Cao. 2019a. “Vision-based detection and visualization of dynamic workspaces.” Autom. Constr. 104 (Aug): 1–13. https://doi.org/10.1016/j.autcon.2019.04.001.
Luo, X., H. Li, X. Yang, Y. Yu, and D. Cao. 2019b. “Capturing and understanding workers’ activities in far-field surveillance videos with deep action recognition and Bayesian nonparametric learning.” Comput.-Aided Civ. Infrastruct. Eng. 34 (4): 333–351. https://doi.org/10.1111/mice.12419.
Luttmann, A., M. Jäger, B. Griefahn, G. Caffier, and F. Liebers. 2003. Preventing musculoskeletal disorders in the workplace. Geneva: World Health Organization.
MassirisFernández, M., J. Álvaro Fernández, J. M. Bajo, and C. A. Delrieux. 2020. “Ergonomic risk assessment based on computer vision and machine learning.” Comput. Ind. Eng. 149 (Nov): 106816. https://doi.org/10.1016/j.cie.2020.106816.
McAtamney, L., and E. Nigel Corlett. 1993. “RULA: A survey method for the investigation of work-related upper limb disorders.” Appl. Ergon. 24 (2): 91–99. https://doi.org/10.1016/0003-6870(93)90080-S.
Memarzadeh, M., A. Heydarian, M. Golparvar-Fard, and J. C. Niebles. 2012. “Real-time and automated recognition and 2D tracking of construction workers and equipment from site video streams.” In Proc., Computing in Civil Engineering, 429–436. Reston, VA: ASCE. https://doi.org/10.1061/9780784412343.0054.
Mitropoulos, P., G. Cupido, and M. Namboodiri. 2009. “Cognitive approach to construction safety: Task demand-capability model.” J. Constr. Eng. Manage. 135 (9): 881–889. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000060.
Nath, N. D., A. H. Behzadan, and S. G. Paal. 2020. “Deep learning for site safety: Real-time detection of personal protective equipment.” Autom. Constr. 112 (Apr): 103085. https://doi.org/10.1016/j.autcon.2020.103085.
Paltrinieri, N., L. Comfort, and G. Reniers. 2019. “Learning about risk: Machine learning for risk assessment.” Saf. Sci. 118 (Oct): 475–486. https://doi.org/10.1016/j.ssci.2019.06.001.
Park, M.-W., and I. Brilakis. 2012. “Construction worker detection in video frames for initializing vision trackers.” Autom. Constr. 28 (Dec): 15–25. https://doi.org/10.1016/j.autcon.2012.06.001.
Park, M.-W., N. Elsafty, and Z. Zhu. 2015. “Hardhat-wearing detection for enhancing on-site safety of construction workers.” J. Constr. Eng. Manage. 141 (9): 04015024. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000974.
Pavllo, D., C. Feichtenhofer, D. Grangier, and M. Auli. 2019. “3D human pose estimation in video with temporal convolutions and semi-supervised training.” In Proc., 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 7745–7754. New York: IEEE.
Poh, C. Q., C. U. Ubeynarayana, and Y. M. Goh. 2018. “Safety leading indicators for construction sites: A machine learning approach.” Autom. Constr. 93 (Sep): 375–386. https://doi.org/10.1016/j.autcon.2018.03.022.
Ray, S. J., and J. Teizer. 2012. “Real-time construction worker posture analysis for ergonomics training.” Adv. Eng. Inf. 26 (2): 439–455. https://doi.org/10.1016/j.aei.2012.02.011.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster r-CNN: Towards real-time object detection with region proposal networks.” In Vol. 1 of Proc., 28th Int. Conf. on Neural Information Processing Systems, 91–99. Cambridge, MA: MIT Press.
Roberts, D., W. T. Calderon, S. Tang, and M. Golparvar-Fard. 2020. “Vision-based construction worker activity analysis informed by body posture.” J. Comput. Civ. Eng. 34 (4): 04020017. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000898.
Rozenfeld, O., R. Sacks, Y. Rosenfeld, and H. Baum. 2010. “Construction job safety analysis.” Saf. Sci. 48 (4): 491–498. https://doi.org/10.1016/j.ssci.2009.12.017.
Sacks, R., O. Rozenfeld, and Y. Rosenfeld. 2009. “Spatial and temporal exposure to safety hazards in construction.” J. Constr. Eng. Manage. 135 (8): 726–736. https://doi.org/10.1061/(ASCE)0733-9364(2009)135:8(726).
Seo, J., S. Han, S. Lee, and H. Kim. 2015. “Computer vision techniques for construction safety and health monitoring.” Adv. Eng. Inf. 29 (2): 239–251. https://doi.org/10.1016/j.aei.2015.02.001.
Shrestha, K., P. P. Shrestha, D. Bajracharya, and E. A. Yfantis. 2015. “Hard-hat detection for construction safety visualization.” J. Constr. Eng. 2015 (1): 1–8. https://doi.org/10.1155/2015/721380.
Tang, S., D. Roberts, and M. Golparvar-Fard. 2020. “Human-object interaction recognition for automatic construction site safety inspection.” Autom. Constr. 120 (Dec): 103356. https://doi.org/10.1016/j.autcon.2020.103356.
Wu, J., N. Cai, W. Chen, H. Wang, and G. Wang. 2019. “Automatic detection of hardhats worn by construction personnel: A deep learning approach and benchmark dataset.” Autom. Constr. 106 (Oct): 102894. https://doi.org/10.1016/j.autcon.2019.102894.
Xiong, R., Y. Song, H. Li, and Y. Wang. 2019. “Onsite video mining for construction hazards identification with visual relationships.” Adv. Eng. Inf. 42 (Oct): 100966. https://doi.org/10.1016/j.aei.2019.100966.
Yan, X., H. Li, H. Zhang, and T. M. Rose. 2018. “Personalized method for self-management of trunk postural ergonomic hazards in construction rebar ironwork.” Adv. Eng. Inf. 37 (Aug): 31–41. https://doi.org/10.1016/j.aei.2018.04.013.
Yan, X., H. Zhang, and H. Li. 2019. “Estimating worker-centric 3D spatial crowdedness for construction safety management using a single 2D camera.” J. Comput. Civ. Eng. 33 (5): 04019030. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000844.
Yang, J., M.-W. Park, P. A. Vela, and M. Golparvar-Fard. 2015. “Construction performance monitoring via still images, time-lapse photos, and video streams: Now, tomorrow, and the future.” Adv. Eng. Inf. 29 (2): 211–224. https://doi.org/10.1016/j.aei.2015.01.011.
Yang, J., Z. Shi, and Z. Wu. 2016. “Vision-based action recognition of construction workers using dense trajectories.” Adv. Eng. Inf. 30 (3): 327–336. https://doi.org/10.1016/j.aei.2016.04.009.
Yu, Y., X. Yang, H. Li, X. Luo, H. Guo, and Q. Fang. 2019. “Joint-level vision-based ergonomic assessment tool for construction workers.” J. Constr. Eng. Manage. 145 (5): 04019025. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001647.
Zhang, H., X. Yan, and H. Li. 2018. “Ergonomic posture recognition using 3D view-invariant features from single ordinary camera.” Autom. Constr. 94 (Oct): 1–10. https://doi.org/10.1016/j.autcon.2018.05.033.
Zhang, M., R. Shi, and Z. Yang. 2020a. “A critical review of vision-based occupational health and safety monitoring of construction site workers.” Saf. Sci. 126 (Jun): 104658. https://doi.org/10.1016/j.ssci.2020.104658.
Zhang, M., M. Zhu, and X. Zhao. 2020b. “Recognition of high-risk scenarios in building construction based on image semantics.” J. Comput. Civ. Eng. 34 (4): 04020019. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000900.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 35 • Issue 6 • November 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Jan 4, 2021
Accepted: Apr 1, 2021
Published online: Aug 12, 2021
Published in print: Nov 1, 2021
Discussion open until: Jan 12, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.