Multisensory and BIM-Integrated Digital Twin to Improve Urban Excavation Safety
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 5
Abstract
Urban excavation is an indispensable process for many construction activities such as road paving, house building, and pipe rehabilitation. However, the everincreasing complexity of underground utilities (e.g., water mains, gas lines, and sewage pipes) in urban environments challenges the safety of urban excavation, posing tremendous risks of potential collision and damage accidents. By obtaining real-time excavation information and high-fidelity simulation to evaluate safety risks, digital twin (DT) has an unexplored potential to improve urban excavation safety (UES). This research aims to investigate how a DT for urban excavation can be developed and used to improve UES. First, a multisensory solution is proposed to equip the physical excavators with the capability to precisely estimate their three-dimensional (3D) poses based on the kinematic model and social spider algorithm (SSA). Second, a building information model (BIM) of buried utilities and a 3D model of the excavators are integrated to form a dynamic virtual model that mirrors the actual excavation process. Third, based on the physical-virtual coupling DT, a real-time safety control method is proposed to proactively monitor urban excavation, dynamically assess collision risk, and timely warn against unsafe behaviors. A system prototype was developed and applied in a case study in Shandong, China. Results show that the system can precisely twin the pose of the excavator, increasing the estimation accuracy of the translation by at least 4.0 cm. The system can display the dynamic spatial position of the excavator and the buried pipes in 3D and automatically guide the excavator to operate safely in real-time, thereby avoiding potential collision accidents.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are available from the corresponding author by request. It mainly includes the following data: (1) field data for pose estimation; and (2) code for algorithm optimization and pose estimation.
Acknowledgments
This research was supported by the National Natural Science Foundation of China (No. 52279136).
References
Adhvaryyu, S., and P. K. Adhvaryyu. 2020. “Application of bio-inspired social spider algorithm in multi-area economic emission dispatch of solar, wind and CHP-based power system.” Soft Comput. 24 (13): 9611–9624. https://doi.org/10.1007/s00500-019-04468-4.
Akanmu, A. A., C. J. Anumba, and O. O. Ogunseiju. 2021. “Towards next generation cyber-physical systems and digital twins for construction.” J. Inf. Technol. Constr. 26 (Jul): 505–525. https://doi.org/10.36680/j.itcon.2021.027.
Assadzadeh, A., M. Arashpour, I. Brilakis, T. Ngo, and E. Konstantinou. 2022. “Vision-based excavator pose estimation using synthetically generated datasets with domain randomization.” Autom. Constr. 134 (Feb): 104089. https://doi.org/10.1016/j.autcon.2021.104089.
Behzadan, A. H., S. Dong, and V. R. Kamat. 2015. “Augmented reality visualization: A review of civil infrastructure system applications.” Adv. Eng. Inf. 29 (2): 252–267. https://doi.org/10.1016/j.aei.2015.03.005.
Croatti, A., M. Gabellini, S. Montagna, and A. Ricci. 2020. “On the integration of agents and digital twins in healthcare.” J. Med. Syst. 44 (9): 1–8. https://doi.org/10.1007/s10916-020-01623-5.
El-bages, M. S., and W. T. Elsayed. 2017. “Social spider algorithm for solving the transmission expansion planning problem.” Electr. Power Syst. Res. 143 (Feb): 235–243. https://doi.org/10.1016/j.epsr.2016.09.002.
Feng, C., V. R. Kamat, and H. Cai. 2018. “Camera marker networks for articulated machine pose estimation.” Autom. Constr. 96 (Dec): 148–160. https://doi.org/10.1016/j.autcon.2018.09.004.
Golparvar-Fard, M., A. Heydarian, and J. C. Niebles. 2013. “Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers.” Adv. Eng. Inf. 27 (4): 652–663. https://doi.org/10.1016/j.aei.2013.09.001.
Gülmez, B., and S. Kulluk. 2019. “Social spider algorithm for training artificial neural networks.” Int. J. Bus. Anal. 6 (4): 32–49. https://doi.org/10.4018/IJBAN.2019100103.
Hosamo, H. H., A. Imran, J. Cardenas-Cartagena, P. R. Svennevig, K. Svidt, and H. K. Nielsen. 2022. “A review of the digital twin technology in the AEC-FM industry.” In Advances in civil engineering, 1–17. Cairo, Egypt: Hindawi. https://doi.org/10.1155/2022/2185170.
Huang, S., K. Yang, H. Xiao, P. Han, J. Qiu, L. Peng, D. Liu, and K. Luo. 2022. “A new head pose tracking method based on stereo visual SLAM.” J. Visual Commun. Image Represent. 82 (Jan): 103402. https://doi.org/10.1016/j.jvcir.2021.103402.
Jiang, W., L. Ding, and C. Zhou. 2022. “Digital twin: Stability analysis for tower crane hoisting safety with a scale model.” Autom. Constr. 138 (Jun): 104257. https://doi.org/10.1016/j.autcon.2022.104257.
Kaewunruen, S., and Q. Lian. 2019. “Digital twin aided sustainability-based lifecycle management for railway turnout systems.” J. Cleaner Prod. 228 (Aug): 1537–1551. https://doi.org/10.1016/j.jclepro.2019.04.156.
Li, S., H. Cai, and V. R. Kamat. 2015. “Uncertainty-aware geospatial system for mapping and visualizing underground utilities.” Autom. Constr. 53 (Apr): 105–119. https://doi.org/10.1016/j.autcon.2015.03.011.
Liang, C., K. M. Lundeen, W. McGee, C. C. Menassa, S. Lee, and V. R. Kamat. 2019. “A vision-based marker-less pose estimation system for articulated construction robots.” Autom. Constr. 104 (Aug): 80–94. https://doi.org/10.1016/j.autcon.2019.04.004.
Liu, B., R. Bao, and F. Sui. 2021. “A fatigue damage-cumulative model in peridynamics.” Chin. J. Aeronaut. 34 (2): 329–342. https://doi.org/10.1016/j.cja.2020.09.046.
Lundeen, K. M., S. Dong, N. Fredricks, M. Akula, J. Seo, and V. R. Kamat. 2016. “Optical marker-based end effector pose estimation for articulated excavators.” Autom. Constr. 65 (May): 51–64. https://doi.org/10.1016/j.autcon.2016.02.003.
Luo, H., M. Wang, P. K. Wong, and J. C. P. Cheng. 2020. “Full body pose estimation of construction equipment using computer vision and deep learning techniques.” Autom. Constr. 110 (Feb): 103016. https://doi.org/10.1016/j.autcon.2019.103016.
Madubuike, O. C., C. J. Anumba, and R. Khallaf. 2022. “A review of digital twin applications in construction.” J. Inf. Technol. Constr. 27 (8): 145–172. https://doi.org/10.36680/j.itcon.2022.008.
Mahmood, B., S. Han, and J. Seo. 2022. “Implementation experiments on convolutional neural network training using synthetic images for 3D pose estimation of an excavator on real images.” Autom. Constr. 133 (Jan): 103996. https://doi.org/10.1016/j.autcon.2021.103996.
Moon, S., and J. Seo. 2017. “Virtual graphic representation of construction equipment for developing a 3D earthwork BIM.” J. Civ. Eng. Manage. 23 (8): 977–984. https://doi.org/10.3846/13923730.2017.1348981.
Mostafa, K., and T. Hegazy. 2021. “Review of image-based analysis and applications in construction.” Autom. Constr. 122 (Feb): 103516. https://doi.org/10.1016/j.autcon.2020.103516.
Panchal, K., and H. Shah. 2013. “3d face recognition based on pose correction using Euler angle method.” In Proc., Int. Conf. on Machine Intelligence and Research Advancement, 467–471. New York: IEEE.
Perumal, L. 2014. “Euler angles: Conversion of arbitrary rotation sequences to specific rotation sequence.” Comput. Anim. Virtual Worlds 25 (5–6): 521–529. https://doi.org/10.1002/cav.1529.
Shao, G., and M. Helu. 2020. “Framework for a digital twin in manufacturing: Scope and requirements.” Manuf. Lett. 24 (Apr): 105–107. https://doi.org/10.1016/j.mfglet.2020.04.004.
Sharafat, A., M. S. Khan, K. Latif, W. A. Tanoli, W. Park, and J. Seo. 2021. “BIM-GIS-based integrated framework for underground utility management system for earthwork operations.” Appl. Sci. 11 (12): 5721. https://doi.org/10.3390/app11125721.
Soltani, M. M., Z. Zhu, and A. Hammad. 2016. “Automated annotation for visual recognition of construction resources using synthetic images.” Autom. Constr. 62 (Feb): 14–23. https://doi.org/10.1016/j.autcon.2015.10.002.
Soltani, M. M., Z. Zhu, and A. Hammad. 2017. “Skeleton estimation of excavator by detecting its parts.” Autom. Constr. 82 (Oct): 1–15. https://doi.org/10.1016/j.autcon.2017.06.023.
Su, X., S. Talmaki, H. Cai, and V. R. Kamat. 2013. “Uncertainty-aware visualization and proximity monitoring in urban excavation: A geospatial augmented reality approach.” Visualization Eng. 1 (1): 1–13. https://doi.org/10.1186/2213-7459-1-2.
Sun, D., C. Ji, S. Jang, S. Lee, J. No, C. Han, J. Han, and M. Kang. 2020. “Analysis of the position recognition of the bucket tip according to the motion measurement method of excavator boom, stick and bucket.” Sensors 20 (10): 2881. https://doi.org/10.3390/s20102881.
Sun, J., and S. Lu. 2020. “An improved single shot multibox for video-rate head pose prediction.” IEEE Sens. J. 20 (20): 12326–12333. https://doi.org/10.1109/JSEN.2020.2999625.
Sung, H., N. Jung, S. Huang, and J. Kim. 2019. “Application of social spider algorithm to optimize train energy.” J. Electr. Eng. Technol. 14 (1): 519–526. https://doi.org/10.1007/s42835-018-00016-6.
Talmaki, S., and V. R. Kamat. 2012. “Real-time hybrid virtuality for prevention of excavation related utility strikes.” J. Comput. Civ. Eng. 28 (3): 04014001. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000269.
Talmaki, S., V. R. Kamat, and H. Cai. 2013. “Geometric modeling of geospatial data for visualization-assisted excavation.” Adv. Eng. Inf. 27 (2): 283–298. https://doi.org/10.1016/j.aei.2013.01.004.
Tang, J., H. Luo, W. Chen, P. K. Wong, and J. C. P. Cheng. 2022. “IMU-based full-body pose estimation for construction machines using kinematics modeling.” Autom. Constr. 138 (Jun): 104217. https://doi.org/10.1016/j.autcon.2022.104217.
Tanoli, W. A., J. W. Seo, A. Sharafat, and S. S. Lee. 2018. “3D design modeling application in machine guidance system for earthwork operations.” KSCE J. Civ. Eng. 22 (12): 4779–4790. https://doi.org/10.1007/s12205-018-0309-y.
Tanoli, W. A., A. Sharafat, J. Park, and J. W. Seo. 2019. “Damage prevention for underground utilities using machine guidance.” Autom. Constr. 107 (Nov): 102893. https://doi.org/10.1016/j.autcon.2019.102893.
Vahdatikhaki, F., A. Hammad, and H. Siddiqui. 2015. “Optimization-based excavator pose estimation using real-time location systems.” Autom. Constr. 56 (Aug): 76–92. https://doi.org/10.1016/j.autcon.2015.03.006.
Wu, S., L. Hou, G. K. Zhang, and H. Chen. 2022. “Real-time mixed reality-based visual warning for construction workforce safety.” Autom. Constr. 139 (Jul): 104252. https://doi.org/10.1016/j.autcon.2022.104252.
Xu, J., and H. Yoon. 2019. “Vision-based estimation of excavator manipulator pose for automated grading control.” Autom. Constr. 98 (Feb): 122–131. https://doi.org/10.1016/j.autcon.2018.11.022.
Yu, J. J. Q., and V. O. K. Li. 2015. “A social spider algorithm for global optimization.” Appl. Soft Comput. 30 (Jan): 614–627. https://doi.org/10.1016/j.asoc.2015.02.014.
Zeng, Y., and H. Guo. 2017. “An Euler angle calculation method for Tailsitter UAV.” In Proc., Chinese Automation Congress (CAC), 814–819. New York: IEEE.
Zhao, J., Y. Hu, and M. Tian. 2021. “Pose estimation of excavator manipulator based on monocular vision marker system.” Sensors 21 (13): 4478. https://doi.org/10.3390/s21134478.
Zou, J., and H. Kim. 2007. “Using hue, saturation, and value color space for hydraulic excavator idle time analysis.” J. Comput. Civ. Eng. 21 (4): 238–246. https://doi.org/10.1061/(ASCE)0887-3801(2007)21:4(238).
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 5 • September 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Jan 30, 2023
Accepted: May 5, 2023
Published online: Jul 4, 2023
Published in print: Sep 1, 2023
Discussion open until: Dec 4, 2023
ASCE Technical Topics:
- Architectural engineering
- Building information modeling
- Building management
- Buried pipes
- Business management
- Construction engineering
- Construction methods
- Dynamic models
- Engineering fundamentals
- Equipment and machinery
- Excavation
- Infrastructure
- Models (by type)
- Pipeline systems
- Pipes
- Practice and Profession
- Probe instruments
- Public administration
- Public health and safety
- Safety
- Three-dimensional models
- Urban and regional development
- Urban areas
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.