Real-Time Struck-By Hazards Detection System for Small- and Medium-Sized Construction Sites Based on Computer Vision Using Far-Field Surveillance Videos
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 6
Abstract
Small- and medium-sized construction sites are not efficiently managed because of the lack of budget and labor in safety management. In this paper, a real-time struck-by hazards detection system based on computer vision is proposed to automatically detect onsite hazards at small- and medium-sized construction sites by analyzing far-field surveillance videos. The proposed system consists of image processing technologies such as object detection, object tracking, image classification, and projective transformation, considering actual small- and medium-sized construction site conditions. Images obtained from small- and medium-sized construction sites have a fixed scene, far-field conditions, and crowded characteristics. In the object detection, an object class suitable for far-field conditions was defined, and object tracking and class changes were applied to ensure field applicability. In addition, to apply projective transformation using fixed scene images, the representative point of the detected object was established. Moreover, for the real-time application of the entire system, appropriate models for each function were selected, and an optimized application process was presented. Consequently, the integrated system, which simultaneously performs hardhat-wearing detection, heavy-equipment operation detection, signal-worker arrangement detection, and heavy-equipment proximity detection, was developed. The performance of the proposed model was evaluated individually, and the feasibility of the proposed system was verified by attaching the qualitative results of the field application to the real small- and medium-sized construction sites. In the quantitative results of the hardhat-wearing detection part, an accuracy of 91% and the system robustness change according to the parameters were presented.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some data, models, or code that support the findings of this study are available from the corresponding author on reasonable request, including (1) samples of the image data used in the experiment; and (2) related information about the object detection model and code (Unity).
Acknowledgments
This work is supported by the Korea Agency for Infrastructure Technology Advancement (KAIA) grant funded by the Ministry of Land, Infrastructure, and Transport (Grant RS-2020-KA156208).
References
Antwi-Afari, M. F., H. Li, J. Seo, and A. Y. L. Wong. 2018. “Automated detection and classification of construction workers’ loss of balance events using wearable insole pressure sensors.” Autom. Constr. 96 (Dec): 189–199. https://doi.org/10.1016/j.autcon.2018.09.010.
Avola, D., L. Cinque, A. Diko, A. Fagioli, G. L. Foresti, A. Mecca, D. Pannone, and C. Piciarelli. 2021. “MS-Faster R-CNN: Multi-stream backbone for improved Faster R-CNN object detection and aerial tracking from UAV images.” Remote Sens. 13 (9): 1670. https://doi.org/10.3390/rs13091670.
BLS (Bureau of Labor Statistics). 2021. “National census of fatal occupational injuries in 2020.” Accessed December 16, 2022. https://www.bls.gov/news.release/pdf/cfoi.pdf.
Broström, M. 2020. “Real-time multi-object tracker using YOLOv5 and deep sort.” Accessed March 15, 2022. https://github.com/mikel-brostrom/Yolov5_DeepSort_Pytorch.
Cheng, T., J. Teizer, G. C. Migliaccio, and U. C. Gatti. 2013. “Automated task-level activity analysis through fusion of real time location sensors and worker’s thoracic posture data.” Autom. Constr. 29 (Jan): 24–39. https://doi.org/10.1016/j.autcon.2012.08.003.
Chi, S., and C. H. Caldas. 2012. “Image-based safety assessment: Automated spatial safety risk identification of earthmoving and surface mining activities.” J. Constr. Eng. Manage. 138 (3): 341–351. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000438.
Dong, S., H. Li, and Q. Yin. 2018. “Building information modeling in combination with real time location systems and sensors for safety performance enhancement.” Saf. Sci. 102 (Feb): 226–237. https://doi.org/10.1016/j.ssci.2017.10.011.
Dubrofsky, E. 2007. Homography estimation. Vancouver, BC, Canada: Univ. of Vancouver.
Fang, Q., H. Li, X. Luo, L. Ding, H. Luo, T. M. Rose, and W. An. 2018a. “Detecting non-hardhat-use by a deep learning method from far-field surveillance videos.” Autom. Constr. 85 (Jan): 1–9. https://doi.org/10.1016/j.autcon.2017.09.018.
Fang, Q., H. Li, X. Luo, L. Ding, T. M. Rose, W. An, and Y. Yu. 2018b. “A deep learning-based method for detecting non-certified work on construction sites.” Adv. Eng. Inf. 35 (Jan): 56–68. https://doi.org/10.1016/j.aei.2018.01.001.
Fang, W., L. Ding, H. Luo, and P. E. D. Love. 2018c. “Falls from heights: A computer vision-based approach for safety harness detection.” Autom. Constr. 91 (Jul): 53–61. https://doi.org/10.1016/j.autcon.2018.02.018.
Fang, W., B. Zhong, N. Zhao, P. E. D. Love, H. Luo, J. Xue, and S. Xu. 2019. “A deep learning-based approach for mitigating falls from height with computer vision: Convolutional neural network.” Adv. Eng. Inf. 39 (Jan): 170–177. https://doi.org/10.1016/j.aei.2018.12.005.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 580–587. New York: IEEE.
Han, S., and S. Lee. 2013. “A vision-based motion capture and recognition framework for behavior-based safety management.” Autom. Constr. 35 (Nov): 131–141. https://doi.org/10.1016/j.autcon.2013.05.001.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Jeelani, I., K. Asadi, H. Ramshankar, K. Han, and A. Albert. 2021. “Real-time vision-based worker localization & hazard detection for construction.” Autom. Constr. 121 (Jan): 103448. https://doi.org/10.1016/j.autcon.2020.103448.
Jocher, G., K. Nishimura, T. Mineeva, and R. Vilariño. 2020. “YOLOv5.” Accessed March 15, 2022. https://github.com/ultralytics/yolov5.
Kelm, A., L. Laußat, A. Meins-Becker, D. Platz, M. J. Khazaee, A. M. Costin, M. Helmus, and J. Teizer. 2013. “Mobile passive radio frequency identification (RFID) portal for automated and rapid control of personal protective equipment (PPE) on construction sites.” Autom. Constr. 36 (Dec): 38–52. https://doi.org/10.1016/j.autcon.2013.08.009.
Kim, D., M. Liu, S. Lee, and V. R. Kamat. 2019. “Remote proximity monitoring between mobile construction resources using camera-mounted UAVs.” Autom. Constr. 99 (Mar): 168–182. https://doi.org/10.1016/j.autcon.2018.12.014.
Kim, H., K. Kim, and H. Kim. 2016. “Vision-based object-centric safety assessment using fuzzy inference: Monitoring struck-by accidents with moving objects.” J. Comput. Civ. Eng. 30 (4): 04015075. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000562.
Kim, K., H. Kim, and H. Kim. 2017. “Image-based construction hazard avoidance system using augmented reality in wearable device.” Autom. Constr. 83 (Nov): 390–403. https://doi.org/10.1016/j.autcon.2017.06.014.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2017. “Imagenet classification with deep convolutional neural networks.” Commun. ACM 60 (6): 84–90. https://doi.org/10.1145/3065386.
Kuhn, H. W. 1955. “The Hungarian method for the assignment problem.” Nav. Res. Logist. Q. 2 (1–2): 83–97. https://doi.org/10.1002/nav.3800020109.
Lee, W., K. Y. Lin, E. Seto, and G. C. Migliaccio. 2017. “Wearable sensors for monitoring on-duty and off-duty worker physiological status and activities in construction.” Autom. Constr. 83 (Nov): 341–353. https://doi.org/10.1016/j.autcon.2017.06.012.
Li, H., M. Lu, S.-C. Hsu, M. Gray, and T. Huang. 2015. “Proactive behavior-based safety management for construction safety improvement.” Saf. Sci. 75 (Jun): 107–117. https://doi.org/10.1016/j.ssci.2015.01.013.
Milan, A., L. Leal-Taixé, I. Reid, S. Roth, and K. Schindler. 2016. “MOT16: A benchmark for multi-object tracking.” Preprint, submitted March 2, 2016. http://arxiv.org/abs/1603.00831.
Ministry of Employment and Labor of Korea. 2020. “Industrial accident status analysis.” Accessed March 15, 2022. https://www.moel.go.kr/info/publict/publictDataView.do?bbs_seq=20211201900.
Nath, N. D., A. H. Behzadan, and S. G. Paal. 2020. “Deep learning for site safety: Real-time detection of personal protective equipment.” Autom. Constr. 112 (Apr): 103085. https://doi.org/10.1016/j.autcon.2020.103085.
Sandler, M., A. Howard, M. Zhu, A. Zhmoginov, and L. C. Chen. 2018. “Mobilenetv2: Inverted residuals and linear bottlenecks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 4510–4520. New York: IEEE.
Seo, J., S. Han, S. Lee, and H. Kim. 2015. “Computer vision techniques for construction safety and health monitoring.” Adv. Eng. Inf. 29 (2): 239–251. https://doi.org/10.1016/j.aei.2015.02.001.
Son, H., H. Seong, H. Choi, and C. Kim. 2019. “Real-time vision-based warning system for prevention of collisions between workers and heavy equipment.” J. Comput. Civ. Eng. 33 (5): 04019029. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000845.
Tang, S., D. Roberts, and M. Golparvar-Fard. 2020. “Human-object interaction recognition for automatic construction site safety inspection.” Autom. Constr. 120 (Dec): 103356. https://doi.org/10.1016/j.autcon.2020.103356.
Teizer, J., B. S. Allread, and U. Mantripragada. 2010. “Automating the blind spot measurement of construction equipment.” Autom. Constr. 19 (4): 491–501. https://doi.org/10.1016/j.autcon.2009.12.012.
Unity Technologies. 2022. “Unity please supply the year of publication.” Accessed March 15, 2022. https://unity.com/.
Wang, L., L. Xie, P. Yang, Q. Deng, S. Du, and L. Xu. 2020. “Hardhat-wearing detection based on a lightweight convolutional neural network with multi-scale features and a top-down module.” J. Sens. 20 (7): 1868. https://doi.org/10.3390/s20071868.
Welch, G., and G. Bishop. 1995. An introduction to the Kalman filter, 127–132. Chapel Hill, NC: Univ. of North Carolina.
Wojke, N., A. Bewley, and D. Paulus. 2017. “Simple online and realtime tracking with a deep association metric.” In Proc., 2017 IEEE Int. Conf. on Image Processing (ICIP), 3645–3649. New York: IEEE.
Wu, J., N. Cai, W. Chen, H. Wang, and G. Wang. 2019. “Automatic detection of hardhats worn by construction personnel: A deep learning approach and benchmark dataset.” Autom. Constr. 106 (Oct): 102894. https://doi.org/10.1016/j.autcon.2019.102894.
Yu, Y., H. Guo, Q. Ding, H. Li, and M. Skitmore. 2017. “An experimental study of real-time identification of construction workers’ unsafe behaviors.” Autom. Constr. 82 (Oct): 193–206. https://doi.org/10.1016/j.autcon.2017.05.002.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 6 • November 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Oct 20, 2022
Accepted: Jun 6, 2023
Published online: Aug 4, 2023
Published in print: Nov 1, 2023
Discussion open until: Jan 4, 2024
ASCE Technical Topics:
- Business management
- Computer vision and image processing
- Computing in civil engineering
- Construction engineering
- Construction management
- Construction sites
- Detection methods
- Disaster risk management
- Engineering fundamentals
- Geotechnical engineering
- Geotechnical investigation
- Methodology (by type)
- Occupational safety
- Practice and Profession
- Project management
- Public administration
- Public health and safety
- Risk management
- Safety
- Site investigation
- Tracking
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.