Seeing through Disaster Rubble in 3D with Ground-Penetrating Radar and Interactive Augmented Reality for Urban Search and Rescue
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 5
Abstract
First responders often lack information and visual clues regarding interior spaces in disaster rubble, preventing efficient, effective, and safe search and rescue for victims trapped in collapsed structures. Rapidly detecting and acquiring information about the voids in collapsed structures that could contain surviving victims is critical for urban search and rescue. However, reconstructing the buried voids in three dimensions (3D) and communicating the relevant information such as buried depth and void size to first responders remain significant challenges. In response, this study proposes a see-through technique by integrating ground-penetrating radar (GPR) with interactive augmented reality (AR). The contribution of this study is twofold. First, a new method is developed to process collected GPR data to reconstruct potential voids in disaster rubble in 3D and extract the buried depth and void size from the GPR data. The coordinates of void boundaries are extracted from multiple GPR scans to generate sparse point clouds. An improved alpha-shape method is exploited to reconstruct the 3D space beneath disaster rubble from the point clouds. Second, an interactive augmented reality interface is developed to enable first responders to visualize the voids in collapsed structures in 3D together with relevant information to assist urban search and rescue. The results from simulations and pilot experiments demonstrate the feasibility and potential of the proposed methods.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or codes that support the findings of this study are provided by the corresponding author upon reasonable request.
Acknowledgments
This research was funded by the National Science Foundation (NSF) via Grants 1850008 and 2129003 and the Tennessee Department of Transportation (TDOT) via the Research Project RES2021-05: “Drones and Other Technologies to Assist in Disaster Relief Efforts.” The authors gratefully acknowledge the support from NSF and TDOT. Any opinions, findings, conclusions, and recommendations expressed in this paper are those of the authors and do not necessarily reflect the views of NSF, TDOT, The University of Tennessee, Knoxville, Loughborough University, The University of Florida, and The University of Texas at San Antonio.
References
Aboudourib, A., M. Serhir, and D. Lesselier. 2019. “3D reconstruction of tree roots under heterogeneous soil conditions using Ground Penetrating Radar.” In Proc., 10th Int. Workshop on Advanced Ground Penetrating Radar, 1–6. Houten, Netherlands: European Association of Geoscientists and Engineers.
Al-Tamimi, M. S. H., G. Sulong, and I. L. Shuaib. 2015. “Alpha shape theory for 3D visualization and volumetric measurement of brain tumor progression using magnetic resonance images.” Magn. Reson. Imaging 33 (6): 787–803. https://doi.org/10.1016/j.mri.2015.03.008.
Alyassin, A. M., J. L. Lancaster, J. H. Downs III, and P. T. Fox. 1994. “Evaluation of new algorithms for the interactive measurement of surface area and volume.” Med. Phys. 21 (6): 741–752. https://doi.org/10.1118/1.597333.
Bacim, F., E. D. Ragan, C. Stinson, S. Scerbo, and D. A. Bowman. 2012. “Collaborative navigation in virtual search and rescue.” In Proc., IEEE Symposium on 3D User Interfaces 2012, 3DUI 2012, 187–188. New York: IEEE.
Bloch, T., R. Sacks, and O. Rabinovitch. 2016. “Interior models of earthquake damaged buildings for search and rescue.” Adv. Eng. Inf. 30 (1): 65–76. https://doi.org/10.1016/j.aei.2015.12.001.
Burian, F., L. Zalud, P. Kocmanova, T. Jilek, L. Kopecny, and L. Kopecny. 2014. “Multi-robot system for disaster area exploration.” WIT Trans. Ecol. Environ. 184: 263–274.
Cai, J., J. Jeon, H. Cai, and S. Li. 2020. “Fusing heterogeneous information for underground utility map generation based on Dempster-Shafer theory.” J. Comput. Civ. Eng. 34 (3): 04020013. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000892.
Campos, A., N. Correia, T. Romão, I. Nunes, and M. Simões-Marques. 2019. “Mobile augmented reality techniques for emergency response.” In Proc., 16th EAI Int. Conf. on Mobile and Ubiquitous Systems: Computing, Networking and Services, 31–39. New York: Association for Computing Machinery.
Carmigniani, J., and B. Furht. 2011. “Augmented reality: An overview.” In Handbook of augmented reality, 3–46. Berlin: Springer.
Chen, J., S. Li, D. Liu, and X. Li. 2020. “AiRobSim: Simulating a multisensor aerial robot for urban search and rescue operation and training.” Sensors 20 (18): 5223. https://doi.org/10.3390/s20185223.
Cist, D. B. 2009. “Non-destructive evaluation after destruction: Using ground penetrating radar for search and rescue.” In Proc., 7th Int. Symp. on Non-Destructive Testing in Civil Engineering, Nantes, France, Expanded Abstracts. Nantes, France: Laboratoire central des ponts et chaussées.
Coovert, M. D., T. Lee, I. Shindev, and Y. Sun. 2014. “Spatial augmented reality as a method for a mobile robot to communicate intended movement.” Comput. Hum. Behav. 34 (May): 241–248. https://doi.org/10.1016/j.chb.2014.02.001.
Couch, S. R. 2008. “Handbook of disaster research.” In Contemporary sociology: A journal of reviews, handbooks of sociology and social research, edited by H. Rodríguez, W. Donner, and J. E. Trainor. Cham, Switzerland: Springer.
de Ravé, E. G., et al. 2016. “DiedricAR: A mobile augmented reality system designed for the ubiquitous descriptive geometry learning.” Multimedia Tools Appl. 75 (16): 9641–9663. https://doi.org/10.1007/s11042-016-3384-4.
Doroodgar, B., Y. Liu, and G. Nejat. 2014. “A learning-based semi-autonomous controller for robotic exploration of unknown disaster scenes while searching for victims.” IEEE Trans. Cybern. 44 (12): 2719–2732. https://doi.org/10.1109/TCYB.2014.2314294.
Frantz, T., B. Jansen, J. Duerinck, and J. Vandemeulebroucke. 2018. “Augmenting Microsoft’s HoloLens with vuforia tracking for neuronavigation.” Healthcare Technol. Lett. 5 (5): 221–225. https://doi.org/10.1049/htl.2018.5079.
Fruehauf, F., A. Heilig, M. Schneebeli, W. Fellin, and O. Scherzer. 2009. “Experiments and algorithms to detect snow avalanche victims using airborne ground-penetrating radar.” IEEE Trans. Geosci. Remote Sens. 47 (7): 2240–2251. https://doi.org/10.1109/TGRS.2009.2012717.
Gianni, M., G. Gonnelli, A. Sinha, M. Menna, and F. Pirri. 2013. “An augmented reality approach for trajectory planning and control of tracked vehicles in rescue environments.” In Proc., 2013 IEEE Int. Symp. on Safety, Security, and Rescue Robotics, SSRR 2013. New York: IEEE.
Gomes, L., O. R. P. Bellon, and L. Silva. 2014. “3D reconstruction methods for digital preservation of cultural heritage: A survey.” Pattern Recognition Lett. 50 (Dec): 3–14. https://doi.org/10.1016/j.patrec.2014.03.023.
Hajebi, K., Y. Abbasi-Yadkori, H. Shahbazi, and H. Zhang. 2011. “Fast approximate nearest-neighbor search with k-nearest neighbor graph.” In Proc., 22nd Int. Joint Conf. on Artificial Intelligence. San Francisco: The International Joint Conferences on Artificial Intelligence.
Heilig, A., M. Schneebeli, and W. Fellin. 2008. “Feasibility study of a system for airborne detection of avalanche victims with ground penetrating radar and a possible automatic location algorithm.” Cold Reg. Sci. Technol. 51 (2–3): 178–190. https://doi.org/10.1016/j.coldregions.2007.06.003.
Hu, D., F. Hou, J. Blakely, and S. Li. 2020a. “Augmented reality based visualization for concrete bridge deck deterioration characterized by ground penetrating radar.” In Proc., Construction Research Congress 2020: Computer Applications–Selected Papers from the Construction Research Congress 2020, 1156–1164. Reston, VA: ASCE.
Hu, D., F. Hou, and S. Li. 2020b. “Ground-penetrating radar-based root architecture detection and characterization.” In Proc., 18th Int. Conf. on Ground Penetrating Radar, 243–246. Houston: Society of Exploration Geophysicists.
Hu, D., S. Li, J. Chen, and V. R. Kamat. 2019. “Detecting, locating, and characterizing voids in disaster rubble for search and rescue.” Adv. Eng. Inf. 42 (Oct): 100974. https://doi.org/10.1016/j.aei.2019.100974.
Kong, C., A. Ferworn, E. Coleshill, J. Tran, and K. G. Derpanis. 2016. “What is a hole? Discovering access holes in disaster rubble with functional and photometric attributes.” J. Field Rob. 33 (6): 825–836. https://doi.org/10.1002/rob.21590.
Levatti, H. U., P. Prat Catalán, A. Ledesma Villalba, A. Cuadrado Cabello, and J. A. Cordero Arias. 2017. “Experimental analysis of 3D cracking in drying soils using ground-penetrating radar.” Geotech. Test. J. 40 (2): 1–23. https://doi.org/10.1520/GTJ20160066.
Li, H., C. Chou, L. Fan, B. Li, D. Wang, and D. Song. 2020. “Toward automatic subsurface pipeline mapping by fusing a ground-penetrating radar and a camera.” IEEE Trans. Autom. Sci. Eng. 17 (2): 722–734. https://doi.org/10.1109/TASE.2019.2941848.
Li, S., H. Cai, and V. R. Kamat. 2015. “Uncertainty-aware geospatial system for mapping and visualizing underground utilities.” Autom. Constr. 53 (May): 105–119. https://doi.org/10.1016/j.autcon.2015.03.011.
Liu, H., Z. Chen, H. Lu, F. Han, C. Liu, J. Li, and J. Cui. 2020a. “Migration of ground penetrating radar with antenna radiation pattern correction.” In Proc., IEEE Geoscience and Remote Sensing Letters. New York: IEEE.
Liu, L., Z. Liu, H. Xie, B. Barrowes, and A. C. Bagtzoglou. 2014. “Numerical simulation of UWB impulse radar vital sign detection at an earthquake disaster site.” Ad Hoc Networks 13 (Part A): 34–41. https://doi.org/10.1016/j.adhoc.2012.08.006.
Liu, X., J. Chen, J. R. Butnor, G. Qin, X. Cui, B. Fan, H. Lin, and L. Guo. 2020b. “Noninvasive 2D and 3D mapping of root zone soil moisture through the detection of coarse roots with ground-penetrating radar.” Water Resour. Res. 56 (5): e2019WR026930. https://doi.org/10.1029/2019WR026930.
Murphy, R. R., J. Casper, and M. Micire. 2001. Potential tasks and research issues for mobile robots in RoboCup rescue. Berlin: Springer.
Nadkarni, P. 2016. “Core technologies: Data mining and ‘big data’.” In Clinical research computing: A practitioner’s handbook, 187–204. Amsterdam, Netherlands: Elsevier.
Núñez-Nieto, X., M. Solla, A. Novo, and H. Lorenzo. 2014. “Three-dimensional ground-penetrating radar methodologies for the characterization and volumetric reconstruction of underground tunneling.” Constr. Build. Mater. J. 71 (Nov): 551–560. https://doi.org/10.1016/j.conbuildmat.2014.08.083.
Otsu, N. 1979. “A threshold selection method from gray-level histograms.” IEEE Trans. Syst. Man Cybern. 9 (1): 62–66. https://doi.org/10.1109/TSMC.1979.4310076.
Poteyeva, M., M. Denver, L. E. Barsky, and B. E. Aguirre. 2007. Search and rescue activities in disasters. New York: Springer.
PTC (Parametric Technology Corporation). 2019. “Industrial augmented reality: Vuforia.” Accessed August 28, 2019. https://www.ptc.com/en/products/augmented-reality.
Reardon, C., K. Lee, and J. Fink. 2018. “Come see this! Augmented reality to enable human-robot cooperative search.” In Proc., 2018 IEEE Int. Symp. on Safety, Security, and Rescue Robotics, SSRR 2018. New York: IEEE.
Tong, Z., J. Gao, and H. Zhang. 2017. “Recognition, location, measurement, and 3D reconstruction of concealed cracks using convolutional neural networks.” Constr. Build. Mater. 146 (Aug): 775–787. https://doi.org/10.1016/j.conbuildmat.2017.04.097.
Vassell, M., O. Apperson, P. Calyam, J. Gillis, and S. Ahmad. 2016. “Intelligent dashboard for augmented reality based incident command response co-ordination.” In Proc., 2016 13th IEEE Annual Consumer Communications & Networking Conf. (CCNC), 976–979. New York: IEEE.
Walte, O., and K. Kostack. 2017. Blender and Bullet physics engine based on fast on-site assessment tool. Vantaa, Finland: Laurea Univ. of Applied Sciences.
Wang, R., H. Lu, J. Xiao, Y. Li, and Q. Qiu. 2018. “The design of an augmented reality system for urban search and rescue.” Accessed December 19, 2019. https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=8535823.
Wani, A. R., S. Shabir, and R. Naaz. 2013. “Augmented reality for fire & emergency services.” In Proc., Int. Conf. on Recent Trends in Communication and Computer Networks, 32–41. Thiruvananthapuram, India: Institute of Doctors Engineers and Scientists.
Warren, C., A. Giannopoulos, and I. Giannakis. 2016. “gprMax: Open source software to simulate electromagnetic wave propagation for Ground Penetrating Radar.” Comput. Phys. Commun. 209 (Dec): 163–170. https://doi.org/10.1016/j.cpc.2016.08.020.
Wibisono, J. K., and H.-M. Hang. 2021. “Fined: Fast inference network for edge detection.” In Proc., 2021 IEEE Int. Conf. on Multimedia and Expo (ICME), 1–6. New York: IEEE.
Yan, K., S. Wu, and G. Fang. 2021. “Detection of quasi-static trapped human being using mono-static UWB life-detection radar.” Appl. Sci. 11 (7): 3129. https://doi.org/10.3390/app11073129.
Yang, D., Z. Zhu, and B. Liang. 2019. “Vital sign signal extraction method based on permutation entropy and EEMD algorithm for ultra-wideband radar.” IEEE Access 7: 178879–178890. https://doi.org/10.1109/ACCESS.2019.2958600.
Yuan, C., S. Li, H. Cai, and V. R. Kamat. 2018. “GPR signature detection and decomposition for mapping buried utilities with complex spatial configuration.” J. Comput. Civ. Eng. 32 (4): 04018026. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000764.
Zhang, W., and A. Hoorfar. 2019. “MIMO ground penetrating radar imaging through multilayered subsurface using total variation minimization.” IEEE Trans. Geosci. Remote Sens. 57 (4): 2107–2115. https://doi.org/10.1109/TGRS.2018.2871463.
Zhekov, S. S., O. Franek, and G. F. Pedersen. 2020. “Dielectric properties of common building materials for ultrawideband propagation studies [measurements corner].” IEEE Antennas Propag. Mag. 62 (1): 72–81. https://doi.org/10.1109/MAP.2019.2955680.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 5 • September 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Jul 6, 2021
Accepted: Apr 26, 2022
Published online: Jun 29, 2022
Published in print: Sep 1, 2022
Discussion open until: Nov 29, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.