Data Augmentation for Improving Deep Learning Models in Building Inspections or Postdisaster Evaluation
Publication: Journal of Performance of Constructed Facilities
Volume 35, Issue 4
Abstract
Current building evaluations, whether for occupant safety or insurance appraisal, are conducted primarily via visual inspections performed by certified individuals. These inspections, which often can number in the hundreds of thousands when performed following a disaster, can take weeks to conduct. This time can significantly affect the economic and societal resilience of a community. This paper proposes a framework for the development of unmanned aerial systems (UAS)-driven object detection algorithms for use in automating visual structural inspections. In this framework, domain-specific data augmentation methods are developed and utilized by image-based deep learning models for building inspections. A large, labeled, posthailstorm building evaluation database was developed to train and validate these models. Three data augmentation methods were developed and implemented: background cropping, high-resolution image cropping, and vent cropping. A unique combination of algorithm, novel data augmentations, and ensembling techniques was investigated to increase the performance of the framework. The results demonstrated that the framework can be applied to structural inspections to increase the efficiency and reliability of these assessments while minimizing the risk to human life.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, and code generated during the study are proprietary or confidential in nature and may only be provided upon approval from the funder: all labeled data generated in this work, and all code generated in this work.
Acknowledgments
Research funding and the unlabeled data was provided by the United Services Automobile Association (USAA). The authors gratefully acknowledge the support of USAA. Any opinions, findings, conclusions, and recommendations expressed by the authors in this paper do not necessarily reflect the views of USAA.
References
Abdel-Qader, I., S. Pashaie-Rad, O. Abudayyeh, and S. Yehia. 2006. “PCA-based algorithm for unsupervised bridge crack detection.” Adv. Eng. Software 37 (12): 771–778. https://doi.org/10.1016/j.advengsoft.2006.06.002.
Aghababaei, M., M. Koliou, and S. G. Paal. 2018. “Performance assessment of building infrastructure impacted by the 2017 Hurricane Harvey in the Port Aransas region.” J. Perform. Constr. Facil. 32 (5): 04018069. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001215.
Anantharaman, R., M. Velazquez, and Y. Lee. 2018. “Utilizing mask R-CNN for detection and segmentation of oral diseases.” In Proc., 2018 IEEE Int. Conf. on Bioinformatics and Biomedicine (BIBM), 2197–2204. New York: IEEE.
Arbeláez, P., J. Pont-Tuset, J. T. Barron, F. Marques, and J. Malik. 2014. “Multiscale combinatorial grouping.” In Proc., 2014 IEEE Conf. on Computer Vision and Pattern Recognition, 328–335. New York: IEEE. https://doi.org/10.1109/CVPR.2014.49.
Arnab, A., and P. H. S. Torr. 2017. “Pixelwise instance segmentation with a dynamically instantiated network.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 879–888. New York: IEEE. https://doi.org/10.1109/CVPR.2017.100.
ATC (Applied Technology Council). 1989. Procedures for postearthquake safety evaluations of buildings. ATC-20. Redwood City, CA: ATC.
ATC (Applied Technology Council). 1995. Addendum to the ATC-20 postearthquake building safety evaluation procedures. Redwood City, CA: ATC.
ATC (Applied Technology Council). 2004. Field manual: safety evaluation of buildings after windstorms and floods. ATC-45. Redwood City, CA: ATC.
Bai, M., and R. Urtasun. 2017. “Deep watershed transform for instance segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 2858–2866. New York: IEEE. https://doi.org/10.1109/CVPR.2017.305.
Bartel, J. 2001. “A picture of bridge health.” NTIAC (Nondestr. Test. Inf. Anal. Center) Newsl. 27 (1): 1–4.
Cai, Z., and N. Vasconcelos. 2018. “Cascade R-CNN: Delving into high quality object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 6154–6162. New York: IEEE.
Cha, Y.-J., W. Choi, and O. Büyüköztürk. 2017. “Deep learning-based crack damage detection using convolutional neural networks.” Comput.-Aided Civ. Infrastruct. Eng. 32 (5): 361–378. https://doi.org/10.1111/mice.12263.
Dai, J., J. He, and J. Sun. 2015. “Convolutional feature masking for joint object and stuff segmentation.” In Proc., 2015 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 3992–4000. New York: IEEE. https://doi.org/10.1109/CVPR.2015.7299025.
Dai, J., K. He, and J. Sun. 2016. “Instance-aware semantic segmentation via multi-task network cascades.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 3150–3158. New York: IEEE. https://doi.org/10.1109/CVPR.2016.343.
Dalal, N., and B. Triggs. 2005. “Histograms of oriented gradients for human detection.” In Proc., 2015 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
De Blasiis, M. R., A. Di Benedetto, M. Fiani, and M. Garozzo. 2019. “Assessing the effect of pavement distresses by means of LiDAR technology.” In Computing in civil engineering 2019: Smart cities, sustainability, and resilience, 146–153. Reston, VA: ASCE.
Dollar, P., Z. Tu, P. Perona, and S. Belongie. 2009. “Integral channel features.” In Proc., 2009 British Machine Vision Association. Guildford, UK: British Machine Vision Association.
Erhan, D., C. Szegedy, A. Toshev, and D. Anguelov. 2014. “Scalable object detection using deep neural networks.” In Proc., 2014 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Everingham, M., L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2010. “The PASCAL Visual Object Classes (VOC) challenge.” Int. J. Comput. Vision 88 (2): 303–338. https://doi.org/10.1007/s11263-009-0275-4.
Felzenszwalb, P. F., R. B. Girshick, and D. McAllester. 2010. “Cascade object detection with deformable part models.” In Proc., 2010 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
FEMA. 2008. “National urban search and rescue response system: Structures specialist position description.” Accessed April 28, 2020. https://www.ohtf1.com/applicants/pdf/Structures%20Specialist%20-%20long.pdf.
Fu, C.-Y., W. Liu, A. Ranga, A. Tyagi, and A. C. Berg. 2016. “DSSD: Deconvolutional single shot detector.” Preprint, submitted January 23, 2017. http://arxiv.org/abs/1701.06659.
German, S., and I. Brilakis, and R. DesRoches. 2011. “Automated detection of exposed reinforcement in post-earthquake safety and structural evaluations.” In Proc., 2011 ISEC-6 Modern Methods and Advances in Structural Engineering and Construction Conf. Fargo, ND: ISEC Press.
German, S., I. Brilakis, and R. DesRoches. 2012. “Rapid entropy-based detection and properties measurement of concrete spalling with machine vision for post-earthquake safety assessments.” Adv. Eng. Inf. 26 (4): 846–858. https://doi.org/10.1016/j.aei.2012.06.005.
Girshick, R. 2015. “Fast R-CNN.” In Proc., 2015 Int. Conf. on Computer Vision. New York: IEEE.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., 2014 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Gulgec, N. S., M. Takáč, and S. N. Pakzad. 2019. “Convolutional neural network approach for robust structural damage detection and localization.” J. Comput. Civ. Eng. 33 (3): 04019005. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000820.
Hanover Insurance Group. 2014. “Roof inspection checklist.” Accessed April 28, 2020. https://www.hanover.com/linec/docs/171-0931.pdf.
Hariharan, B., P. Arbeláez, R. Girshick, and J. Malik. 2014. “Simultaneous detection and segmentation.” In Vol. 8695 of Computer vision—ECCV 2014. Lecture notes in computer science, edited by D. Fleet, T. Pajdla, B. Schiele, and T. Tuytelaars. Cham, Switzerland: Springer.
Hariharan, B., P. Arbeláez, R. Girshick, and J. Malik. 2015. “Hypercolumns for object segmentation and fine-grained localization.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 447–456. New York: IEEE. https://doi.org/10.1109/CVPR.2015.7298642.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask R-CNN.” In Proc., 2017 IEEE Int. Conf. on Computer Vision, 2980–2988. New York: IEEE.
He, K., X. Zhang, S. Ren, and J. Sun. 2014. “Spatial pyramid pooling in deep convolutional networks for visual recognition.” In Proc., 2014 European Conf. on Computer Vision. Edinburgh, UK: European Conference on Computer Vision.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Hezaveh, M. M., C. Kanan, and C. Salvaggio. 2017. “Roof damage assessment using deep learning.” In Proc., IEEE Applied Imagery Pattern Recognition Workshop (AIPR), 6403–6408. New York: IEEE. https://doi.org/10.1109/AIPR.2017.8457946.
Hoskere, V., J.-W. Park, H. Yoon, and B. F. Spencer, Jr. 2019. “Vision-based modal survey of civil infrastructure using unmanned aerial vehicles.” J. Struct. Eng. 145 (7): 04019062. https://doi.org/10.1061/(ASCE)ST.1943-541X.0002321.
Huang, J., et al. 2017. “Speed/accuracy trade-offs for modern convolutional object detectors.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 3296–3297. New York: IEEE.
Jahanshahi, M. R., and S. F. Masri. 2012. “A novel vision-based crack quantification approach by incorporating depth perception for condition assessment of structures.” In Computing in Civil Engineering (2012). Reston, VA: ASCE. https://doi.org/10.1061/9780784412343.0067.
Kashani, A. G., A. J. Graettinger, and T. Dao. 2016. “Lidar-based methodology to evaluate fragility models for tornado-induced roof damage.” Nat. Hazards Rev. 17 (3): 04016006. https://doi.org/10.1061/(ASCE)NH.1527-6996.0000224.
Kirillov, A., E. Levinkov, B. Andres, B. Savchynskyy, and C. Rother. 2017. “InstanceCut: From edges to instances with MultiCut.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 7322–7331. New York: IEEE. https://doi.org/10.1109/CVPR.2017.774.
Koch, C., Z. Zhu, S. G. Paal, and I. Brilakis. 2015. “Machine vision techniques for condition assessment of civil infrastructure.” In Integrated imaging and vision techniques for industrial inspection. Advances in computer vision and pattern recognition, edited by Z. Liu, H. Ukida, P. Ramuhalli, and K. Niel. London: Springer.
Krizhevsky, A., I. Sutskever, and G. Hinton. 2012. “ImageNet classification with deep convolutional neural networks.” In Proc., 2012 Conf. on Neural Information Processing Systems. San Diego: NeurIPS.
LeCun, Y., B. Boser, J. S. Denker, D. Henderson, R. E. Howard, W. Hubbard, and L. D. Jackel. 1989. “Backpropagation applied to handwritten zip code recognition.” Neural Comput. 1 (4): 541–551. https://doi.org/10.1162/neco.1989.1.4.541.
Li, Y., H. Qi, J. Dai, X. Ji, and Y. Wei. 2017. “Fully convolutional instance-aware semantic segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 4438–4446. New York: IEEE. https://doi.org/10.1109/CVPR.2017.472.
Lin, T. Y., P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. 2017. “Feature pyramid networks for object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 2117–2125. New York: IEEE.
Liu, L., W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu, and M. Pietikäinen. 2020. “Deep learning for generic object detection: A survey.” Int. J. Comput. Vision 128 (2): 261–318. https://doi.org/10.1007/s11263-019-01247-4.
Liu, S., J. Jia, S. Fidler, and R. Urtasun. 2017. “SGN: Sequential grouping networks for instance segmentation.” In Proc., IEEE Int. Conf. on Computer Vision (ICCV), Venice, 2017, 3516–3524. New York: IEEE. https://doi.org/10.1109/ICCV.2017.378.
Liu, W., D. Anguelov, D. Erhan, C. Szegedy, and S. Reed. 2016. “SSD: Single shot multibox detector.” In Proc., 2016 European Conf. on Computer Vision. Edinburgh, UK: European Conference on Computer Vision.
Liu, Z., A. Shahrel, T. Ohashi, and E. Toshiaki. 2002. “Tunnel crack detection and classification system based on image processing.” In Vol. 4664 of Machine vision applications in industrial inspection X, 145–152. Bellingham, WA: International Society for Optics and Photonics.
Loken, A., C. E. Wittich, L. Brito, and M. K. Saifullah. 2020. “Digital reconnaissance and performance assessment of rural infrastructure for 2018 natural hazards.” J. Perform. Constr. Facil. 34 (4): 04020054. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001460.
Mneymneh, B. E., M. Abbas, and H. Khoury. 2019. “Vision-based framework for intelligent monitoring of hardhat wearing on construction sites.” J. Comput. Civ. Eng. 33 (2): 04018066. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000813.
Paal, S. G., J.-S. Jeon, I. Brilakis, and R. Desroches. 2015. “Automated damage index estimation of reinforced concrete columns for post-earthquake evaluations.” J. Struct. Eng. 141 (9). https://doi.org/10.1061/(ASCE)ST.1943-541X.0001200.
Paal, S. G., J.-S. Jeon, I. Brilakis, and R. DesRoches. 2014. “Automated measurement of concrete spalling through reinforcement detection.” J. Struct. Eng. 141 (9). https://doi.org/10.1061/(ASCE)ST.1943-541X.0001200.
Park, S., S. Bang, H. Kim, and H. Kim. 2019. “Patch-based crack detection in black box images using convolutional neural networks.” J. Comput. Civ. Eng. 33 (3): 04019017. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000831.
Pinheiro, P. O., R. Collobert, and P. Dollar. 2015. “Learning to segment object candidates.” In Proc., 2015 Conf. on Neural Information Processing Systems. San Diego: NeurIPS.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., 2016 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Redmon, J., and A. Farhadi. 2017. “YOLO9000: Better, faster, stronger.” In Proc., 2017 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” In Proc., 2015 Conf. on Neural Information Processing Systems. San Diego: NeurIPS.
Sakhakarmi, S., J. Park, and C. Cho. 2019. “Enhanced machine learning classification accuracy for scaffolding safety using increased features.” J. Constr. Eng. Manage. 145 (2): 04018133. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001601.
Son, H., H. Seong, H. Choi, and C. Kim. 2019. “Real-time vision-based warning system for prevention of collisions between workers and heavy equipment.” J. Comput. Civ. Eng. 33 (5): 04019029. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000845.
Soviany, P., and R. T. Ionescu. 2018. “Optimizing the trade-off between single-stage and two-stage deep object detectors using image difficulty prediction.” In Proc., 2018 20th Int. Symp. on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC). Timisoara, Romania: West Univ. of Timisoara.
Staniek, M. 2017. “Stereo vision method application to road inspection.” Baltic J. Road Bridge Eng. 12 (1): 38–47. https://doi.org/10.3846/bjrbe.2017.05.
Sutley, E. J., K. Vazquez, J. H. Kim, T. Dao, B. Johnston, and J. Hunt. 2020. “Performance of manufactured housing during Hurricanes Irma and Michael.” J. Perform. Constr. Facil. 34 (4): 04020078. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001486.
Texas Department of Insurance. 2019. “Final compilation of Hurricane Harvey data.” Accessed April 28, 2020. https://www.tdi.texas.gov/reports/documents/harvey-dc-final-06302019.pdf.
Tsai, Y., and C. C. Wei. 2019. “Accelerated disaster reconnaissance using automatic traffic sign detection with UAV and AI.” In Computing in civil engineering 2019. Reston, VA: ASCE. https://doi.org/10.1061/9780784482445.052.
Uijlings, J. R. R., K. E. A. van de Sande, T. Gevers, and A. W. M. Smeulders. 2013. “Selective search for object recognition.” Int. J. Comput. Vision 104 (2): 154–171. https://doi.org/10.1007/s11263-013-0620-5.
Vaillant, R., C. Monrocq, and Y. Le Cun. 1994. “Original approach for the localisation of objects in images.” In Proc., 1994 IEEE Proc., on Vision, Image, and Signal Processing, 245. New York: IEEE.
van de Sande, K. E. A., J. R. R. Uijlings, T. Gevers, and A. W. M. Smeulders. 2011. “Segmentation as selective search for object recognition.” In Proc., 2011 Int. Conf. on Computer Vision, 1879–1886. New York: IEEE. https://doi.org/10.1109/ICCV.2011.6126456.
Viola, P., and M. Jones. 2001. “Rapid object detection using a boosted cascade of simple features.” In Proc., 2001 IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Ye, S., S. H. H. Nourzad, A. Pradhan, I. Bartoli, and A. Kontsos. 2014. “Automated detection of damaged areas after hurricane sandy using aerial color images.” In Proc., Computing in Civil and Building Engineering (2014). Reston, VA: ASCE. https://doi.org/10.1061/9780784413616.223.
Yeum, C. M., and S. J. Dyke. 2015. “Vision-based automated crack detection for bridge inspection.” Comput.-Aided Civ. Infrastruct. Eng. 30 (10): 759–770. https://doi.org/10.1111/mice.12141.
Zhang, A., K. C. P. Wang, Y. Fei, Y. Liu, S. Tao, C. Chen, J. Q. Li, and B. Li. 2018a. “Deep learning–based fully automated pavement crack detection on 3D asphalt surfaces with an improved CrackNet.” J. Comput. Civ. Eng. 32 (5): 04018041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000775.
Zhang, K., Y. Zhang, and H. D. Cheng. 2020a. “Self-supervised structure learning for crack detection based on cycle-consistent generative adversarial networks.” J. Comput. Civ. Eng. 34 (3): 04020004. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000883.
Zhang, M., Z. Cao, Z. Yang, and X. Zhao. 2020b. “Utilizing computer vision and fuzzy inference to evaluate level of collision safety for workers and equipment in a dynamic environment.” J. Constr. Eng. Manage. 146 (6): 04020051. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001802.
Zhang, S., L. Wen, X. Bian, Z. Lei, and S. Z. Li. 2018b. “Single-shot refinement neural network for object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 4203–4212. New York: IEEE.
Zhou, Z., J. Gong, and M. Guo. 2016. “Image-based 3D reconstruction for posthurricane residential building damage assessment.” J. Comput. Civ. Eng. 30 (2): 04015015. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000480.
Zhu, Y., C. Zhao, J. Wang, X. Zhao, Y. Wu, and H. Lu. 2017. “CoupleNet: Coupling global structure with local parts for object detection.” In Proc., IEEE Int. Conf. on Computer Vision, 4126–4134. New York: IEEE.
Zhu, Z., and I. Brilakis. 2010. “Machine vision-based concrete surface quality assessment.” J. Constr. Eng. Manage. 136 (2): 210–218. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000126.
Zhu, Z., S. German, and I. Brilakis. 2011. “Visual retrieval of concrete crack properties for automated post-earthquake structural safety evaluation.” Autom. Constr. 20 (7): 874–883. https://doi.org/10.1016/j.autcon.2011.03.004.
Information & Authors
Information
Published In
Journal of Performance of Constructed Facilities
Volume 35 • Issue 4 • August 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Nov 12, 2020
Accepted: Feb 2, 2021
Published online: May 14, 2021
Published in print: Aug 1, 2021
Discussion open until: Oct 14, 2021
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.