Automating Building Damage Reconnaissance to Optimize Drone Mission Planning for Disaster Response
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 3
Abstract
Rapid reconnaissance of building damage is critical for disaster response and recovery. Drones have been utilized to collect aerial images of affected areas in order to assess building damage. However, there are two challenges. First, processing many aerial images to detect and classify building damage based on a consistent standard remains laborious and complex, necessitating a new automated solution to achieve accurate building damage detection and classification. Second, drone operations during disaster response rely primarily on human operators’ experience and seldom use the obtained building damage information to optimize drone mission planning. Therefore, this study proposes a new method, which automates building damage reconnaissance with drone mission planning for disaster response operations. Specifically, a deep learning method is developed to detect and classify building damages using a newly labeled dataset consisting of 24,496 distinct instances of building damage. This deep learning method is validated, achieving 71.9% mean average precision. In addition, building damage information is modeled and integrated into mission planning, in order to optimize drones’ task assignments and route calculations. A tornado disaster in Tennessee is used as a case study to quantitatively evaluate this methodology. The present study concludes that optimal drone mission planning during disaster response can be augmented using accurate building damage information acquired from deep learning methods.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or codes that support the findings of this study are provided by the corresponding author upon reasonable request.
Acknowledgments
This research was funded by the US National Science Foundation (NSF) via Grants 1850008 and 2129003, Tennessee Department of Transportation (TDOT) via the contract RES2021-05 “Drones and Other Technologies to Assist Disaster Relief Efforts,” and Science Alliance at the University of Tennessee Knoxville. The authors gratefully acknowledge the support from NSF, TDOT, and Science Alliance. Any opinions, findings, recommendations, and conclusions in this paper are those of the authors, and do not necessarily reflect the views of NSF, TDOT, The University of Tennessee, Knoxville, The University of Florida, and The University of Texas at San Antonio.
References
Baker, J. W., and C. A. Cornell. 2008. “Uncertainty propagation in probabilistic seismic loss estimation.” Struct. Saf. 30 (3): 236–252. https://doi.org/10.1016/j.strusafe.2006.11.003.
Buratti, N., F. Minghini, E. Ongaretto, M. Savoia, and N. Tullini. 2017. “Empirical seismic fragility for the precast RC industrial buildings damaged by the 2012 Emilia (Italy) earthquakes.” Earthquake Eng. Struct. Dyn. 46 (14): 2317–2335. https://doi.org/10.1002/eqe.2906.
Chang, L., A. S. Elnashai, and B. F. Spencer Jr. 2012. “Post-earthquake modelling of transportation networks.” Struct. Infrastruct. Eng. 8 (10): 893–911.
Chao, I.-M., B. L. Golden, and E. A. Wasil. 1996. “The team orienteering problem.” Eur. J. Oper. Res. 88 (3): 464–474. https://doi.org/10.1016/0377-2217(94)00289-4.
Chen, J., S. Li, D. Liu, and X. Li. 2020. “AiRobSim: Simulating a multisensor aerial robot for urban search and rescue operation and training.” Sensors 20 (18): 5223. https://doi.org/10.3390/s20185223.
Chen, J., S. Li, D. Liu, and W. Lu. 2022. “Indoor camera pose estimation via style-transfer 3D models.” Comput.-Aided Civ. Infrastruct. Eng. 37 (3): 335–353. https://doi.org/10.1111/mice.12714.
Chen, J., W. Lu, and F. Xue. 2021. “‘Looking beneath the surface’: A visual-physical feature hybrid approach for unattended gauging of construction waste composition.” J. Environ. Manage. 286 (Jan): 112233. https://doi.org/10.1016/j.jenvman.2021.112233.
Chen, L., H. Zhang, J. Xiao, L. Nie, J. Shao, W. Liu, and T.-S. Chua. 2017. “Sca-cnn: Spatial and channel-wise attention in convolutional networks for image captioning.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 5659–5667. New York: IEEE.
Chen, S.-C., M. Chen, N. Zhao, S. Hamid, K. Chatterjee, and M. Armella. 2009. “Florida public hurricane loss model: Research in multi-disciplinary system integration assisting government policy making.” Gov. Inf. Q. 26 (2): 285–294. https://doi.org/10.1016/j.giq.2008.12.004.
Cheng, C., A. H. Behzadan, and A. Noshadravan. 2021. “Deep learning for post-hurricane aerial damage assessment of buildings.” Computer-Aided Civ. Infrastruct. Eng. 36 (6): 695–710. https://doi.org/10.1111/mice.12658.
Chowdhury, T., R. Murphy, and M. Rahnemoonfar. 2022. “RescueNet: A high resolution UAV semantic segmentation benchmark dataset for natural disaster damage assessment.” Preprint, submitted February 24, 2022. https://doi.org/10.48550/arXiv.2202.12361.
Cooner, A. J., Y. Shao, and J. B. Campbell. 2016. “Detection of urban damage using remote sensing and machine learning algorithms: Revisiting the 2010 Haiti earthquake.” Rem. Sens. 8 (10): 868. https://doi.org/10.3390/rs8100868.
Cornell, C. A., F. Jalayer, R. O. Hamburger, and D. A. Foutch. 2002. “Probabilistic basis for 2000 SAC federal emergency management agency steel moment frame guidelines.” J. Struct. Eng. 128 (4): 526–533. https://doi.org/10.1061/(ASCE)0733-9445(2002)128:4(526).
Dorigo, M., M. Birattari, and T. Stutzle. 2006. “Ant colony optimization.” IEEE Comput. Intell. Mag. 1 (4): 28–39. https://doi.org/10.1109/MCI.2006.329691.
Doswell, C. A., H. E. Brooks, and N. Dotzek. 2009. “On the implementation of the enhanced Fujita scale in the USA.” Atmos. Res. 93 (1–3): 554–563. https://doi.org/10.1016/j.atmosres.2008.11.003.
Doyle Rice. 2022. “Weather disasters broke records and killed over 600 Americans in 2021.” Accessed January 29, 2022. https://www.usatoday.com/story/news/nation/2022/01/10/weather-2021-death-toll/9157670002/.
Du, A., J. Cai, and S. Li. 2021. “Metamodel-based state-dependent fragility modeling for Markovian sequential seismic damage assessment.” Eng. Struct. 243 (May): 112644. https://doi.org/10.1016/j.engstruct.2021.112644.
Du, A., and J. E. Padgett. 2020. “Investigation of multivariate seismic surrogate demand modeling for multi-response structural systems.” Eng. Struct. 207 (Jun): 110210. https://doi.org/10.1016/j.engstruct.2020.110210.
Du, A., and J. E. Padgett. 2021. “Entropy-based intensity measure selection for site-specific probabilistic seismic risk assessment.” Earthquake Eng. Struct. Dyn. 50 (2): 560–579. https://doi.org/10.1002/eqe.3346.
Du, A., J. E. Padgett, and A. Shafieezadeh. 2020. “Influence of intensity measure selection on simulation-based regional seismic risk assessment.” Earthquake Spectra 36 (2): 647–672. https://doi.org/10.1177/8755293019891717.
FEMA. 2012. Hazus-MH 2.1—Earthquake model technical manual. Washington, DC: FEMA.
Geiß, C., H. Taubenböck, S. Tyagunov, A. Tisch, J. Post, and T. Lakes. 2014. “Assessment of seismic building vulnerability from space.” Earthquake Spectra 30 (4): 1553–1583. https://doi.org/10.1193/121812EQS350M.
Giordano, N., F. de Luca, A. Sextos, F. Ramirez Cortes, C. Fonseca Ferreira, and J. Wu. 2021. “Empirical seismic fragility models for Nepalese school buildings.” Nat. Hazards 105 (1): 339–362. https://doi.org/10.1007/s11069-020-04312-1.
Gramajo, G., and P. Shankar. 2017. “An efficient energy constraint based UAV path planning for search and coverage.” Int. J. Aerosp. Eng. 2017. https://doi.org/10.1155/2017/8085623.
Grogan, S., R. Pellerin, and M. Gamache. 2021. “Using tornado-related weather data to route unmanned aerial vehicles to locate damage and victims.” OR Spectr. 43 (4): 905–939. https://doi.org/10.1007/s00291-021-00640-1.
Guo, M.-H., T.-X. Xu, J.-J. Liu, Z.-N. Liu, P.-T. Jiang, T.-J. Mu, S.-H. Zhang, R. R. Martin, M.-M. Cheng, and S.-M. Hu. 2022. “Attention mechanisms in computer vision: A survey.” In Computational visual media, 1–38. New York: Springer. https://doi.org/10.1007/s41095-022-0271-y.
Gupta, R., B. Goodman, N. Patel, R. Hosfelt, S. Sajeev, E. Heim, J. Doshi, K. Lucas, H. Choset, and M. Gaston. 2019. “Creating xBD: A dataset for assessing building damage from satellite imagery.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops, 10–17. New York: IEEE.
Hayat, S., E. Yanmaz, C. Bettstetter, and T. X. Brown. 2020. “Multi-objective drone path planning for search and rescue with quality-of-service requirements.” Auton. Rob. 44 (7): 1183–1198. https://doi.org/10.1007/s10514-020-09926-9.
He, J., S. Erfani, X. Ma, J. Bailey, Y. Chi, and X.-S. Hua. 2021. “$\alpha $-IoU: A family of power intersection over union losses for bounding box regression.” Adv. Neur. Inf. Process. Syst. 34 (Dec): 20230–20242.
Herbin, A. H., and M. Barbato. 2012. “Fragility curves for building envelope components subject to windborne debris impact.” J. Wind Eng. Ind. Aerodyn. 107 (Feb): 285–298. https://doi.org/10.1016/j.jweia.2012.05.005.
Hou, Q., D. Zhou, and J. Feng. 2021. “Coordinate attention for efficient mobile network design.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 13713–13722. New York: IEEE.
Hu, D., J. Chen, and S. Li. 2022a. “Reconstructing unseen spaces in collapsed structures for search and rescue via deep learning based radargram inversion.” Autom. Constr. 140 (Aug): 104380. https://doi.org/10.1016/j.autcon.2022.104380.
Hu, D., J. Chen, S. Li, and V. R. Kamat. 2019. “Detecting, locating, and characterizing voids in disaster rubbles for search and rescue.” Adv. Eng. Inf. 42 (Oct): 100974 https://doi.org/10.1016/j.aei.2019.100974.
Hu, D., L. Chen, J. Du, J. Cai, and S. Li. 2022b. “Seeing through disaster rubble in 3D with ground-penetrating radar and interactive augmented reality for urban search and rescue.” J. Comput. Civ. Eng. 36 (5): 04022021. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001038.
Hu, D., and S. Li. 2022. “Recognizing object surface materials to adapt robotic disinfection in infrastructure facilities.” Comput.-Aided Civ. Infrastruct. Eng. 37 (12): 1521–1546. https://doi.org/10.1111/mice.12811.
Hu, D., H. Zhong, S. Li, J. Tan, and Q. He. 2020. “Segmenting areas of potential contamination for adaptive robotic disinfection in built environments.” Build. Environ. 184 (Jul): 107226. https://doi.org/10.1016/j.buildenv.2020.107226.
Hu, J., L. Shen, S. Albanie, G. Sun, and E. Wu. 2017. “Squeeze-and-excitation networks.” IEEE Trans. Pattern Anal. Mach. Intell. 42 (8): 2011–2023. https://doi.org/10.1109/TPAMI.2019.2913372.
Huang, Z., C. Chen, and M. Pan. 2020. “Multiobjective UAV path planning for emergency information collection and transmission.” IEEE Int. Things J. 7 (8): 6993–7009. https://doi.org/10.1109/JIOT.2020.2979521.
Jocher, G., et al. 2021. “Ultralytics/yolov5: v6.0 - YOLOv5n ‘Nano’ models, Roboflow integration, TensorFlow export, OpenCV DNN support (v6.0).” Accessed May 18, 2022. https://zenodo.org/record/5563715#.Yobjxy-B27g.
Kakooei, M., and Y. Baleghi. 2017. “Fusion of satellite, aircraft, and UAV data for automatic disaster damage assessment.” Int. J. Remote Sens. 38 (8–10): 2511–2534. https://doi.org/10.1080/01431161.2017.1294780.
Khodaverdizahraee, N., H. Rastiveis, and A. Jouybari. 2020. “Segment-by-segment comparison technique for earthquake-induced building damage map generation using satellite imagery.” Int. J. Disaster Risk Reduct. 46 (Jul): 101505. https://doi.org/10.1016/j.ijdrr.2020.101505.
Kyrkou, C., and T. Theocharides. 2020. “Emergencynet: Efficient aerial image classification for drone-based emergency monitoring using atrous convolutional feature fusion.” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 13: 1687–1699. https://doi.org/10.1109/JSTARS.2020.2969809.
Lei, L. P., L. Y. Liu, L. Zhang, J. T. Bi, Y. H. Wu, Q. Jiao, and W. Zhang. 2010. “Assessment and analysis of collapsing houses by aerial images in the Wenchuan earthquake.” J. Remote Sens. 14 (2): 333–344.
Li, B., S. Patankar, B. Moridian, and N. Mahmoudian. 2018. “Planning large-scale search and rescue using team of UAVs and charging stations.” In Proc., 2018 IEEE Int. Symp. on Safety, Security, and Rescue Robotics, SSRR 2018. New York: IEEE. https://doi.org/10.1109/SSRR.2018.8468631.
Liu, S., L. Qi, H. Qin, J. Shi, and J. Jia. 2018. “Path aggregation network for instance segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 8759–8768. New York: IEEE.
Marshall, T. P., D. McCarthy, J. G. LaDue, J. Wurman, C. R. Alexander, P. Robinson, and K. Kosiba. 2008. “Damage survey and deduction of vortex structure of the Greensburg, KS tornado.” In Proc., 24th Conf. Severe Local Storms. Boston: American Meteorological Society.
McDonald, J. R., K. C. Mehta, D. A. Smith, and J. A. Womble. 2009. “The enhanced Fujita scale: Development and implementation.” In Forensic engineering 2009, 719–728. Reston, VA: ASCE.
Miura, H., T. Aridome, and M. Matsuoka. 2020. “Deep learning-based identification of collapsed, non-collapsed and blue tarp-covered buildings from post-disaster aerial images.” Remote Sens. 12 (12): 1924. https://doi.org/10.3390/rs12121924.
Murphy, R. R., E. Steimle, C. Griffin, C. Cullins, M. Hall, and K. Pratt. 2008. “Cooperative use of unmanned sea surface and micro aerial vehicles at Hurricane Wilma.” J. Field Rob. 25 (3): 164–180. https://doi.org/10.1002/rob.20235.
Nedjati, A., G. Izbirak, B. Vizvari, and J. Arkat. 2016. “Complete coverage path planning for a multi-UAV response system in post-earthquake assessment.” Robotics 5 (4): 26. https://doi.org/10.3390/robotics5040026.
Nielson, B. G., and R. DesRoches. 2007. “Seismic fragility methodology for highway bridges using a component level approach.” Earthquake Eng. Struct. Dyn. 36 (6): 823–839. https://doi.org/10.1002/eqe.655.
Padgett, J. E., and R. DesRoches. 2008. “Methodology for the development of analytical fragility curves for retrofitted bridges.” Earthquake Eng. Struct. Dyn. 37 (8): 1157–1174. https://doi.org/10.1002/eqe.801.
Pi, Y., N. D. Nath, and A. H. Behzadan. 2020. “Convolutional neural networks for object detection in aerial imagery for disaster response and recovery.” Adv. Eng. Inf. 43 (Jun): 101009. https://doi.org/10.1016/j.aei.2019.101009.
Pinelli, J.-P., G. Pita, K. Gurley, B. Torkian, S. Hamid, and C. Subramanian. 2011. “Damage characterization: Application to Florida public hurricane loss model.” Nat. Hazard. Rev. 12 (4): 190–195. https://doi.org/10.1061/(ASCE)NH.1527-6996.0000051.
Poli, R., J. Kennedy, and T. Blackwell. 2007. “Particle swarm optimization.” Swarm Intell. 1 (1): 33–57. https://doi.org/10.1007/s11721-007-0002-0.
Qi, J., D. Song, H. Shang, N. Wang, C. Hua, C. Wu, X. Qi, and J. Han. 2016. “Search and rescue rotary-wing UAV and its application to the Lushan Ms 7.0 earthquake.” J. Field Rob. 33 (3): 290–321. https://doi.org/10.1002/rob.21615.
Rahnemoonfar, M., T. Chowdhury, A. Sarkar, D. Varshney, M. Yari, and R. R. Murphy. 2021. “Floodnet: A high resolution aerial imagery dataset for post flood scene understanding.” IEEE Access 9 (Jul): 89644–89654. https://doi.org/10.1109/ACCESS.2021.3090981.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 779–788. New York: IEEE.
Roueche, D. B., F. T. Lombardo, and D. O. Prevatt. 2017. “Empirical approach to evaluating the tornado fragility of residential structures.” J. Struct. Eng. 143 (9): 04017123. https://doi.org/10.1061/(ASCE)ST.1943-541X.0001854.
Schaefer, M., R. Teeuw, S. Day, D. Zekkos, P. Weber, T. Meredith, and C. J. van Westen. 2020. “Low-cost UAV surveys of hurricane damage in Dominica: Automated processing with co-registration of pre-hurricane imagery for change analysis.” Nat. Hazards 101 (3): 755–784. https://doi.org/10.1007/s11069-020-03893-1.
Vaneev, A. 2021. “BITEOPT - derivative-free optimization method.” Accessed May 18, 2022. https://github.com/avaneev/biteopt.
Van Huynh, D., T. Do-Duy, L. D. Nguyen, M. T. Le, N. S. Vo, and T. Q. Duong. 2022. “Real-Time optimized path planning and energy consumption for data collection in unmanned ariel vehicles-aided intelligent wireless sensing.” IEEE Trans. Ind. Inf. 18 (4): 2753–2761. https://doi.org/10.1109/TII.2021.3114358.
Vickery, P. J., F. J. Masters, M. D. Powell, and D. Wadhera. 2009. “Hurricane hazard modeling: The past, present, and future.” J. Wind Eng. Ind. Aerodyn. 97 (7–8): 392–405. https://doi.org/10.1016/j.jweia.2009.05.005.
Wang, C.-Y., H.-Y. M. Liao, Y.-H. Wu, P.-Y. Chen, J.-W. Hsieh, and I.-H. Yeh. 2020. “CSPNet: A new backbone that can enhance learning capability of CNN.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops, 390–391. New York: IEEE.
Whitley, D. 1994. “A genetic algorithm tutorial.” Stat. Comput. 4 (2): 65–85. https://doi.org/10.1007/BF00175354.
Xie, S., J. Duan, S. Liu, Q. Dai, W. Liu, Y. Ma, R. Guo, and C. Ma. 2016. “Crowdsourcing rapid assessment of collapsed buildings early after the earthquake based on aerial remote sensing image: A case study of Yushu earthquake.” Remote Sens. 8 (9): 759. https://doi.org/10.3390/rs8090759.
Xu, F., Y. Zhang, R. Wang, C. Mi, Y. Zhang, Y. Huang, and J. Yang. 2021. “Heuristic path planning method for multistatic UAV-Borne SAR imaging system.” IEEE J. Sel. Top. Appl. Earth Obs. Remote Sens. 14 (Aug): 8522–8536. https://doi.org/10.1109/JSTARS.2021.3106449.
Yamazaki, F., Y. Yano, and M. Matsuoka. 2005. “Visual damage interpretation of buildings in Bam city using QuickBir-d images following the 2003 Bam, Iran, earthquake.” Supplement, Earthquake Spectra 21 (S1): 329–336. https://doi.org/10.1193/1.2101807.
Yu, X., C. Li, and J. F. Zhou. 2020. “A constrained differential evolution algorithm to solve UAV path planning in disaster scenarios.” Knowl.-Based Syst. 204 (Jun): 106209. https://doi.org/10.1016/j.knosys.2020.106209.
Zhang, R., H. Li, K. Duan, S. You, K. Liu, F. Wang, and Y. Hu. 2020. “Automatic detection of earthquake-damaged buildings by integrating UAV oblique photography and infrared thermal imaging.” Remote Sens. 12 (16): 2621. https://doi.org/10.3390/rs12162621.
Zhu, X., J. Liang, and A. Hauptmann. 2021. “MSNet: A multilevel instance segmentation network for natural disaster damage assessment in aerial videos.” In Proc., IEEE Winter Conf. on Applications of Computer Vision (WACV), 2022–2031. New York: IEEE.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 3 • May 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Feb 3, 2022
Accepted: Aug 4, 2022
Published online: Feb 1, 2023
Published in print: May 1, 2023
Discussion open until: Jul 1, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Da Hu, Sultan Al Shafian, Segmentation and Tracking of Moving Objects on Dynamic Construction Sites, Construction Research Congress 2024, 10.1061/9780784485262.007, (60-69), (2024).
- Da Hu, Tien Yee, Integrated Framework for Bridge Crack Detection and Semantic BIM Model Generation Using Drone-Captured Imagery and Deep Learning Techniques, ASCE Inspire 2023, 10.1061/9780784485163.020, (167-175), (2023).