Exploring the Generalizability of Deep Convolutional Neural Networks for Post-Hurricane Damage Assessment
Publication: Lifelines 2022
ABSTRACT
Research on artificial intelligence (AI) and aerial imaging technologies has led to new opportunities in large-scale damage assessment for disaster management. As in other domains, the quality of AI predictions in disaster damage assessment primarily depends on the size, quality, and heterogeneity of training data. However, each disaster leaves behind a unique visual pattern of destruction that is influenced by both the intrinsic properties of the event (e.g., wind speed of a hurricane, ground acceleration of an earthquake) and the characteristics of the location it is taking place (e.g., building portfolio, terrain, vegetation cover). This makes the development and assessment of AI models for universal damage assessment extremely challenging, as the size and feature space of data collected from each disaster is distinctive. This study aims to evaluate and benchmark the generalizability of a damage assessment AI model across different hurricane events. The model is a stacked convolutional neural network (CNN) that is initially trained on aerial videos from Hurricane Dorian (2019) but is tested without retraining on unseen videos of several hurricanes in the 2017 season, including Harvey, Maria, and Irma. The testing accuracy and the square of Earth Mover’s Distance (EMD2) values are used to evaluate the model generalizability. Overall, the model achieves 36.7% precision in detecting buildings and 43.1% accuracy in damage level classification. Additionally, EMD2 is found to be negatively correlated with accuracy. Results also indicate that the model generally has a better performance in predicting extreme ground truth labels (i.e., no damaged or destroyed buildings). This behavior is expected and consistent with how human annotators label buildings for damage since humans find these extreme cases more straightforward to label than cases where buildings have suffered intermediate damage levels.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Barrington, L., Ghosh, S., Greene, M., Har-Noy, S., Berger, J., Gill, S., Lin, A. Y.-M., and Huyck, C. (2012). “Crowdsourcing earthquake damage assessment using remote sensing imagery.” Annals of Geophysics, 54(6).
Blake, E. S. (2018). “The 2017 Atlantic hurricane season: catastrophic losses and costs.” Weatherwise, 71(3), 28–37.
Cheng, C.-S., Behzadan, A. H., and Noshadravan, A. (2021). “Deep Learning for Post-Hurricane Aerial Damage Assessment of Buildings.” Computer-Aided Civil and Infrastructure Engineering, 36(6), 695–710.
Chollet, F. (2017). Deep Learning with Python. Manning Publications Company.
Chung, Y., Haas, P. J., Upfal, E., and Kraska, T. (2018). “Unknown examples & machine learning model generalization.” arXiv preprint arXiv:.08294.
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. (2010). “The Pascal Visual Object Classes (VOC) Challenge.” International Journal of Computer Vision, 88(2), 303–338.
FEMA. (2003). Multi-hazard Loss Estimation Methodology: Hurricane Model HAZUS-MH MR3, Technical Manual.
FEMA. (2020). FEMA Preliminary Damage Assessment Guide, Federal Emergency Management Agency.
Halverson, J. B. (2018). “The Costliest Hurricane Season in U.S. History.” Weatherwise, 71(2), 20–27.
Hou, L., Yu, C.-P., and Samaras, D. (2016). “Squared Earth Mover's Distance-based Loss for Training Deep Neural Networks.” ArXiv preprint arXiv:1611.05916.
Hurt, B., Kligerman, S., and Hsiao, A. (2020). “Deep Learning Localization of Pneumonia: 2019 Coronavirus (COVID-19) Outbreak.” Journal of Thoracic Imaging, 35(3), W87–W89.
Kawaguchi, K., Kaelbling, L. P., and Bengio, Y. (2017). “Generalization in deep learning.” arXiv preprint arXiv:.05468.
Khajwal, A. B., and Noshadravan, A. (2021). “An Uncertainty-aware Framework for Reliable Disaster Damage Assessment Via Crowdsourcing.” International Journal of Disaster Risk Reduction, 102110.
Lue, E., Wilson, J. P., and Curtis, A. (2014). “Conducting Disaster Damage Assessments with Spatial Video, Experts, and Citizens.” Applied Geography, 52, 46–54.
Nex, F., Duarte, D., Tonolo, F. G., and Kerle, N. (2019). “Structural Building Damage Detection with Deep Learning: Assessment of a State-of-the-art CNN in Operational Conditions.” Remote Sensing, 11(23), 2765.
Neyshabur, B., Bhojanapalli, S., McAllester, D., and Srebro, N. (2017). “Exploring generalization in deep learning.” Advances in neural information processing systems, 5947–5956.
Paschali, M., Conjeti, S., Navarro, F., and Navab, N. (2018). Generalizability vs. robustness: investigating medical imaging networks using adversarial examples, Springer.
Paul, H. Y., Kim, T. K., and Lin, C. T. (2020). “Generalizability of deep learning tuberculosis classifier to COVID-19 chest radiographs: new tricks for an old algorithm?” Journal of thoracic imaging, 35(4), W102–W104.
Pi, Y., Nath, N. D., and Behzadan, A. H. (2020). “Convolutional Neural Networks for Object Detection in Aerial Imagery for Disaster Response and Recovery.” Advanced Engineering Informatics, 43, 101009.
Rubner, Y., Tomasi, C., and Guibas, L. J. (2000). “The Earth Mover's Distance as A Metric for Image Retrieval.” International journal of computer vision, 40(2), 99–121.
Xie, S., Duan, J., Liu, S., Dai, Q., Liu, W., Ma, Y., Guo, R., and Ma, C. (2016). “Crowdsourcing Rapid Assessment of Collapsed Buildings Early after the Earthquake Based on Aerial Remote Sensing Image: A Case Study of Yushu Earthquake.” Remote Sensing, 8(9), 759.
Yeum, C., Dyke, S., Benes, B., Hacker, T., Ramirez, J., Lund, A., and Pujol, S. (2017). “Rapid, automated image classification for documentation.” Proc., 7th Conf. on Advances in Experimental Structural Engineering.
Yeum, C. M., Dyke, S. J., Benes, B., Hacker, T., Ramirez, J., Lund, A., and Pujol, S. J. J. O. P. O. C. F. (2019). “Postevent reconnaissance image documentation using automated classification.” 33(1), 04018103.
Yoshida, Y., and Miyato, T. (2017). “Spectral norm regularization for improving the generalizability of deep learning.” arXiv preprint arXiv:.10941.
Zhang, C., Bengio, S., Hardt, M., Recht, B., and Vinyals, O. (2016). “Understanding deep learning requires rethinking generalization.” arXiv preprint arXiv:.03530.
Information & Authors
Information
Published In
Lifelines 2022
Pages: 577 - 585
History
Published online: Nov 16, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.