Dynamic Hazardous Proximity Zone Design for Excavator Based on 3D Mechanical Arm Pose Estimation via Computer Vision
Publication: Journal of Construction Engineering and Management
Volume 149, Issue 7
Abstract
In the construction industry, collision accidents between excavators and workers can result in a significant number of fatalities. Hence, low-cost supervision systems are necessary to monitor the motion status of excavators and extract the dynamic hazardous proximity zones around these excavators. Recent computer vision–based methods are convenient and economical, and they also do not require the installation of additional sensors on excavators; however, most of these methods only focus on two-dimensional (2D) or static hazardous proximity zones, which are excessively rough for the fine management required at construction sites. Therefore, a framework for the three-dimensional (3D) dynamic hazardous proximity zone design of excavators based on real-time 3D pose estimation is proposed herein. For this purpose, a set of excavator 3D pose estimation algorithms containing the 2D keypoint detection module and 2D-to-3D lifting module were trained based on two corresponding data sets, one of which was collected on the Internet and the other was generated via Cinema4D software. The real-time motion status was calculated based on the 3D poses estimated from the video after postprocessing and coordinate transformation. Moreover, trajectories could be predicted to form the 3D dynamic hazardous proximity zone for an excavator. Notably, this study developed a computer vision–based pipeline for sensor-free excavator 3D pose estimation and applied the 3D pose estimation algorithm to a specific management issue, thus providing a reference for excavator–worker collision prevention and transforming the conventional static and general onsite construction safety management to a dynamic and specific one.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This research was supported by the Science Research Plan of Shanghai Municipal Science and Technology Committee (Grant No. 20dz1201301), and the 2021 Science Research Plan of Shanghai Housing and Urban-Rural Development Management Committee (Grant No. 2021-002-4049).
References
Assadzadeh, A., M. Arashpour, I. Brilakis, T. Ngo, and E. Konstantinou. 2022. “Vision-based excavator pose estimation using synthetically generated datasets with domain randomization.” Autom. Constr. 134 (Feb): 104089. https://doi.org/10.1016/j.autcon.2021.104089.
Carion, N., F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. 2020. “End-to-end object detection with transformers.” In Proc., European Conf. on Computer Vision, 213–229. New York: Springer.
Chen, Y., Z. Wang, Y. Peng, Z. Zhang, G. Yu, and J. Sun. 2018. “Cascaded pyramid network for multi-person pose estimation.” In Proc., 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 7103–7112. New York: IEEE.
DeVries, T., and G. W. Taylor. 2017. “Improved regularization of convolutional neural networks with cutout.” Preprint, submitted August 15, 2017. https://arxiv.org/abs/1708.04552.
Du, Y., M. C. Dorneich, and B. Steward. 2016. “Virtual operator modeling method for excavator trenching.” Autom. Constr. 70 (Oct): 14–25. https://doi.org/10.1016/j.autcon.2016.06.013.
Fargnoli, M., and M. Lombardi. 2020. “Building information modelling (BIM) to enhance occupational safety in construction activities: Research trends emerging from one decade of studies.” Buildings 10 (6): 98. https://doi.org/10.3390/buildings10060098.
Gallagher, N. B. 2020. Savitzky-Golay smoothing and differentiation filter. Manson, WA: Eigenvector Research Incorporated.
Han, L., W. Mingzhu, P. K. Y. Wong, T. Jingyuan, and J. C. P. Cheng. 2021. “Vision-based pose forecasting of construction equipment for monitoring construction site safety.” In Proc., 18th Int. Conf. on Computing in Civil and Building Engineering: ICCCBE 2020, 1127–1138. New York: Springer International Publishing.
Heinrich, H. W. 1941. “Industrial accident prevention. A scientific approach.” In Industrial accident prevention. A scientific approach. 2nd ed. New York: McGraw-Hill.
Hossain, M. R. I., and J. J. Little. 2018. “Exploiting temporal information for 3D human pose estimation.” In Proc., European Conf. on Computer Vision (ECCV), 68–84. New York: Springer.
HSE (Health and Safety Executive). 2009a. Excavator industry health & safety. London: HSE.
HSE (Health and Safety Executive). 2009b. The safe use of vehicles on construction sites. London: HSE.
Jingyuan, T., L. Han, P. K. Y. Wong, and J. C. P. Cheng. 2021. Study of IMU installation position for posture estimation of excavators. New York: Springer.
Kim, J., S. Chi, and J. Seo. 2018. “Interaction analysis for vision-based activity identification of earthmoving excavators and dump trucks.” Autom. Constr. 87 (Mar): 297–308. https://doi.org/10.1016/j.autcon.2017.12.016.
Kim, K., Y. Cho, and S. Zhang. 2016. “Integrating work sequences and temporary structures into safety planning: Automated scaffolding-related safety hazard identification and prevention in BIM.” Autom. Constr. 70 (Oct): 128–142. https://doi.org/10.1016/j.autcon.2016.06.012.
Kumar, S., H. Gupta, D. Yadav, I. A. Ansari, and O. P. Verma. 2021. “YOLOv4 algorithm for the real-time detection of fire and personal protective equipments at construction sites.” Multimedia Tools Appl. 81 (16): 22163–22183. https://doi.org/10.1007/s11042-021-11280-6.
Li, H., X. Yang, M. Skitmore, F. Wang, and P. Forsythe. 2017. “Automated classification of construction site hazard zones by crowd-sourced integrated density maps.” Autom. Constr. 81 (Sep): 328–339. https://doi.org/10.1016/j.autcon.2017.04.007.
Li, W., H. Liu, R. Ding, M. Liu, P. Wang, and W. Yang. 2022. “Exploiting temporal contexts with strided transformer for 3D human pose estimation.” IEEE Trans. Multimedia 25 (Sep): 1282–1293. https://doi.org/10.1109/TMM.2022.3141231.
Liang, C., K. Lundeen, W. McGee, C. Menassa, S. Lee, and V. Kamat. 2018. “Stacked hourglass networks for markerless pose estimation of articulated construction robots.” In Proc., 35th Int. Symp. on Automation and Robotics in Construction. Waterloo, ON, Canada: International Association for Automation and Robotics in Construction.
Liang, C.-J., K. M. Lundeen, W. McGee, C. C. Menassa, S. Lee, and V. R. Kamat. 2019. “A vision-based marker-less pose estimation system for articulated construction robots.” Autom. Constr. 104 (Aug): 80–94. https://doi.org/10.1016/j.autcon.2019.04.004.
Lundeen, K. M., S. Dong, N. Fredricks, M. Akula, J. Seo, and V. R. Kamat. 2016. “Optical marker-based end effector pose estimation for articulated excavators.” Autom. Constr. 65 (May): 51–64. https://doi.org/10.1016/j.autcon.2016.02.003.
Luo, H., M. Wang, P. K.-Y. Wong, and J. C. P. Cheng. 2020. “Full body pose estimation of construction equipment using computer vision and deep learning techniques.” Autom. Constr. 110 (Feb): 103016. https://doi.org/10.1016/j.autcon.2019.103016.
Mahmood, B., S. Han, and J. Seo. 2022. “Implementation experiments on convolutional neural network training using synthetic images for 3D pose estimation of an excavator on real images.” Autom. Constr. 133 (Jan): 103996. https://doi.org/10.1016/j.autcon.2021.103996.
Messi, L., B. Naticchia, A. Carbonari, L. Ridolfi, and G. M. Di Giuda. 2020. “Development of a digital twin model for real-time assessment of collision hazards.” In Proc., Creative Construction e-Conf. 2020, 14–19. Budapest, Hungary: Budapest Univ. of Technology and Economics.
Moon, H., N. Dawood, and L. Kang. 2014. “Development of workspace conflict visualization system using 4D object of work schedule.” Adv. Eng. Inf. 28 (1): 50–65. https://doi.org/10.1016/j.aei.2013.12.001.
Newell, A., K. Yang, and J. Deng. 2016. “Stacked hourglass networks for human pose estimation.” In Proc., European Conf. on Computer Vision, 483–499. New York: Springer.
OpenMMLab. 2020. “OpenMMLab pose estimation toolbox and benchmark.” Accessed March 10, 2022. https://openmmlab.com.
OSHA (Occupational Safety and Health Administration). 2015. “Safety training and education.” Accessed March 10, 2022. https://www.osha.gov/.
Pegula, S. M. 2013. “An analysis of fatal occupational injuries at road construction sites, 2003-2010.” Monthly Lab. Rev. 136: 1.
Ray, S. J., and J. Teizer. 2013. “Computing 3D blind spots of construction equipment: Implementation and evaluation of an automated measurement and visualization method utilizing range point cloud data.” Autom. Constr. 36 (Dec): 95–107. https://doi.org/10.1016/j.autcon.2013.08.007.
Roberts, D., W. Torres Calderon, S. Tang, and M. Golparvar-Fard. 2020. “Vision-based construction worker activity analysis informed by body posture.” J. Comput. Civ. Eng. 34 (4): 04020017. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000898.
Roberts, D., Y. Wang, A. Sabet, and M. Golparvar-Fard. 2019. “Annotating 2D imagery with 3D kinematically configurable assets of construction equipment for training pose-informed activity analysis and safety monitoring algorithms.” In Proc., ASCE Int. Conf. on Computing in Civil Engineering (i3CE), 32–38. Reston, VA: ASCE.
Savitzky, A., and M. J. Golay. 1964. “Smoothing and differentiation of data by simplified least squares procedures.” Anal. Chem. 36 (8): 1627–1639. https://doi.org/10.1021/ac60214a047.
Seong, H., H. Choi, H. Cho, S. Lee, H. Son, and C. Kim. 2017. “Vision-based safety vest detection in a construction scene.” In Proc., ISARC. Int. Symp. on Automation and Robotics in Construction. Waterloo, ON, Canada: International Association for Automation and Robotics in Construction.
Shen, X., E. Marks, N. Pradhananga, and T. Cheng. 2016. “Hazardous proximity zone design for heavy construction excavation equipment.” J. Constr. Eng. Manage. 142 (6): 1–6. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001108.
Smith, S. W. 1999. The scientist & engineer’s guide to digital signal processing. San Diego, CA: California Technical Publishing.
Soltani, M. M. 2017. “Excavator pose estimation for safety monitoring by fusing computer vision and RTLS data.” Doctoral dissertation, Gina Cody School of Engineering and Computer Science, Concordia Univ.
Soltani, M. M., Z. Zhu, and A. Hammad. 2016. “Towards part-based construction equipment pose estimation using synthetic images.” In Proc., Construction Research Congress, 980–989. Reston, VA: ASCE.
Soltani, M. M., Z. Zhu, and A. Hammad. 2018. “Framework for location data fusion and pose estimation of excavators using stereo vision.” J. Comput. Civ. Eng. 32 (6): 04018045. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000783.
Vaswani, A., N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, Ł. Kaiser, and I. Polosukhin. 2017. “Attention is all you need.” In Proc., Advances in Neural Information Processing Systems, 5998–6008. Red Hook, NY: Curran Associates.
Wang, J., S. Tan, X. Zhen, S. Xu, F. Zheng, Z. He, and L. Shao. 2021a. “Deep 3D human pose estimation: A review.” Comput. Vision Image Understanding 210 (Sep): 103225. https://doi.org/10.1016/j.cviu.2021.103225.
Wang, M., P. Wong, H. Luo, S. Kumar, V. Delhi, and J. Cheng. 2019. “Predicting safety hazards among construction workers and equipment using computer vision and deep learning techniques.” In Proc., ISARC. Int. Symp. on Automation and Robotics in Construction, 399–406. Amsterdam, Netherlands: Elsevier.
Wang, Y., Z. Xu, X. Wang, C. Shen, B. Cheng, H. Shen, and H. Xia. 2021b. “End-to-end video instance segmentation with transformers.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 8741–8750. New York: IEEE.
Wen-Der, Y., L. Hsien-Chou, H. Wen-Ta, C. Hsien-Kuan, T. Chi-Kong, and L. Chen-Chung. 2020. “Automatic safety monitoring of construction hazard working zone: A semantic segmentation based deep learning approach.” In Proc., 2020 the 7th Int. Conf. on Automation and Logistics (ICAL), 54–59. New York: Association for Computing Machinery.
Xiong, R., Y. Song, Y. Wang, and Y. Duan. 2019. “Application of convolutional neural network-based 3D posture estimation in behavioral analysis of construction workers.” China Saf. Sci. J. 29 (7): 64–69. https://doi.org/10.16265/j.cnki.issn1003-3033.2019.07.011.
Yuan, C., S. Li, and H. Cai. 2017. “Vision-based excavator detection and tracking using hybrid kinematic shapes and key nodes.” J. Comput. Civ. Eng. 31 (1): 04016038. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000602.
Yun, S., D. Han, S. J. Oh, S. Chun, J. Choe, and Y. Yoo. 2019. “CutMix: Regularization strategy to train strong classifiers with localizable features.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 6023–6032. New York: IEEE.
Zhao, J., Y. Hu, and M. Tian. 2021. “Pose estimation of excavator manipulator based on monocular vision marker system.” Sensors 21 (13): 4478. https://doi.org/10.3390/s21134478.
Zheng, C., S. Zhu, M. Mendieta, T. Yang, C. Chen, and Z. Ding. 2021. “3D human pose estimation with spatial and temporal transformers.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 11656–11665. New York: IEEE.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 149 • Issue 7 • July 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Aug 25, 2022
Accepted: Mar 2, 2023
Published online: Apr 27, 2023
Published in print: Jul 1, 2023
Discussion open until: Sep 27, 2023
ASCE Technical Topics:
- Business management
- Computer vision and image processing
- Computing in civil engineering
- Construction engineering
- Construction management
- Construction methods
- Disaster risk management
- Disasters and hazards
- Engineering fundamentals
- Excavation
- Infrastructure
- Methodology (by type)
- Occupational safety
- Pipeline management
- Pipeline systems
- Practice and Profession
- Public administration
- Public health and safety
- Safety
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.