PEER Hub ImageNet: A Large-Scale Multiattribute Benchmark Data Set of Structural Images
Publication: Journal of Structural Engineering
Volume 146, Issue 10
Abstract
With the rapid development of machine learning (ML) and deep learning (DL) in computer vision, adopting these learning tools in vision-based structural health monitoring (SHM) and rapid damage assessment is attracting interest in structural engineering. However, several critical issues become impediments, namely, no general automated detection framework, insufficient labeled data, lack of collaboration, and inconsistent research approaches. Thus, in this paper, the authors propose a general automated framework, namely, the Pacific Earthquake Engineering Research (PEER) Hub ImageNet (-Net), in which eight benchmark classification tasks are defined based on domain knowledge and past experience. A large-scale multiattribute -Net data set containing 36,413 pairs of images and labels was established, which was further separated into eight sub-data sets and open-sourced online. These pairwise images and labels can directly contribute to similar classification tasks and the raw structural images can further be used for labeling object localization and segmentation in future studies. Benchmark experiments with various DL models and training strategies were conducted with reported results. Two applications of -Net were further explored, namely, image-based postdisaster assessment of the 1999 Chi-Chi earthquake and the 2018 -Net Challenge. In conclusion, open-sourced data sets and benchmark results are the foundation for future studies where the extension applications reveal the great potential and contribution of -Net in vision-based SHM.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
As the major contribution of this paper, the authors open source the eight mentioned sub-data sets as benchmark data sets in recognizing corresponding structural attributes at https://apps.peer.berkeley.edu/phi-net. Researchers can download these sub-data sets according to posted instructions for noncommercial use. All eight data set files are converted to four-dimensional (4D) tensors (N, 224, 224, 3) for images, and K-dimensional tensors (N, K) for labels corresponding to K classes, where N is the number of data. These tensors are further stored in NumPy array format and compressed to a zip file with the data storage architecture containing X_train.npy, y_train.npy, X_test.npy, and y_test.npy.
Acknowledgments
The authors acknowledge the funding support of the Tsinghua-Berkeley Shenzhen Institute (TBSI), China, and appreciate the assistance from many student volunteers and PEER staff.
References
Amhaz, R., S. Chambon, J. Idier, and V. Baltazart. 2016. “Automatic crack detection on two-dimensional pavement images: An algorithm based on minimal path selection.” IEEE Trans. Intell. Transp. Syst. 17 (10): 2718–2729. https://doi.org/10.1109/TITS.2015.2477675.
Baidu Images. n.d. “Baidu images.” Accessed February 12, 2017. https://image.baidu.com/.
Cha, Y. J., W. Choi, and O. Buyukozturk. 2017. “Deep learning-based crack damage detection using convolutional neural networks.” Comput.-Aided Civ. Infrastruct. Eng. 32 (5): 361–378. https://doi.org/10.1111/mice.12263.
Charron, N., E. McLaughlin, S. Phillips, K. Goorts, S. Narasimhan, and S. L. Waslander. 2019. “Automated bridge inspection using mobile ground robotics.” J. Struct. Eng. 145 (11): 04019137. https://doi.org/10.1061/(ASCE)ST.1943-541X.0002404.
Deng, J., D. Wei, S. Richard, L. Li-Jia, L. Kai, and L. Fei-Fei. 2009. “ImageNet: A large-scale hierarchical image database.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 248–255. New York: IEEE. https://doi.org/10.1109/CVPR.2009.5206848.
DesignSafe. 2020. “Data depot | DesignSafe-CI.” Accessed May 21, 2020. www.designsafe-ci.org/data/browser/public.
Dong, L., and J. Shan. 2013. “A comprehensive review of earthquake-induced building damage detection with remote sensing techniques.” ISPRS J. Photogramm. Remote Sens. 84 (Oct): 85–99. https://doi.org/10.1016/j.isprsjprs.2013.06.011.
Dorafshan, S., R. J. Thomas, and M. Maguire. 2018. “Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete.” Constr. Build. Mater. 186 (Oct): 1031–1045. https://doi.org/10.1016/j.conbuildmat.2018.08.011.
EERI (Earthquake Engineering Research Institute). n.d. “Learning from earthquakes.” Accessed February 12, 2017. https://www.eeri.org/projects/learning-from-earthquakes-lfe/.
Gao, Y., B. Kong, and K. M. Mosalam. 2019. “Deep leaf-bootstrapping generative adversarial network for structural image data augmentation.” Comput.-Aided Civ. Infrastruct. Eng. 34 (9): 755–773. https://doi.org/10.1111/mice.12458.
Gao, Y., K. Li, K. Mosalam, and S. Günay. 2018. “Deep residual network with transfer learning for image-based structural damage recognition.” In Proc., 11th US National Conf. on Earthquake Engineering, Integrating Science, Engineering & Policy. San Francisco: Curran Associates.
Gao, Y., and K. M. Mosalam. 2018. “Deep transfer learning for image-based structural damage recognition.” Comput.-Aided Civ. Infrastruct. Eng. 33 (9): 748–768. https://doi.org/10.1111/mice.12363.
Gao, Y., and K. M. Mosalam. 2019. PEER Hub ImageNet (-Net): A large-scale multi-attribute benchmark dataset of structural images. Berkeley, CA: Pacific Earthquake Engineering Research Center, Univ. of California, Berkeley.
Goodfellow, I., Y. Bengio, and A. Courville. 2016. Deep learning. Cambridge, MA: MIT Press.
Goodfellow, I. J., Y. Bulatov, J. Ibarz, S. Arnoud, and V. Shet. 2013. “Multi-digit number recognition from street view imagery using deep convolutional neural networks.” Preprint, submitted December 20, 2013. https://arxiv.org/abs/1312.6082.
Google. n.d. “Google images.” Accessed February 12, 2017. https://images.google.com/.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Int. Conf. on Computer Vision & Pattern Recognition, 770–778. New York: IEEE. https://doi.org/10.1109/CVPR.2016.90.
Ioffe, S., and C. Szegedy. 2015. “Batch normalization: Accelerating deep network training by reducing internal covariate shift.” Preprint, submitted February 11, 2015. https://arxiv.org/abs/1502.03167.
Kaggle. n.e. “Kaggle: Your machine learning and data science community.” Accessed August 22, 2018. http://www.kaggle.com.
Keras. n.d. “Keras applications.” Accessed May 15, 2018. https://keras.io/applications/.
Khuc, T., and F. N. Catbas. 2018. “Structural identification using computer vision–based bridge health monitoring.” J. Struct. Eng. 144 (2): 04017202. https://doi.org/10.1061/(ASCE)ST.1943-541X.0001925.
Koch, C., K. Georgieva, V. Kasireddy, B. Akinci, and P. Fieguth. 2015. “A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure.” Adv. Eng. Inf. 29 (2): 196–210. https://doi.org/10.1016/j.aei.2015.01.008.
Krizhevsky, A. 2009. Learning multiple layers of features from tiny images. Toronto: Univ. of Toronto.
LeCun, Y., L. Bottou, Y. Bengio, and P. Haffner. 1998. “Gradient-based learning applied to document recognition.” Proc. IEEE 86 (11): 2278–2324. https://doi.org/10.1109/5.726791.
Li, B., and K. M. Mosalam. 2012. “Seismic performance of reinforced-concrete stairways during the 2008 Wenchuan earthquake.” J. Perform. Constr. Facil. 27 (6): 721–730. https://doi.org/10.1061/(ASCE)CF.1943-5509.0000382.
Liang, X. 2019. “Image-based post-disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization.” Comput.-Aided Civ. Infrastruct. Eng. 34 (5): 415–430. https://doi.org/10.1111/mice.12425.
Litjens, G., T. Kooi, B. E. Bejnordi, A. A. A. Setio, F. Ciompi, M. Ghafoorian, J. A. van der Laak, B. van Ginneken, and C. I. Sánchez. 2017. “A survey on deep learning in medical image analysis.” Med. Image Anal. 42 (Dec): 60–88. https://doi.org/10.1016/j.media.2017.07.005.
Maguire, M., S. Dorafshan, and R. J. Thomas. 2018. SDNET2018: A concrete crack image dataset for machine learning applications. Logan, UT: Utah State Univ.
Moehle, J. 2014. Seismic design of reinforced concrete buildings. New York: McGraw-Hill Professional.
Nair, V., and G. E. Hinton. 2010. “Rectified linear units improve restricted Boltzmann machines.” In Proc., 27th Int. Conf. on Machine Learning, 807–814. Madison, WI: Omnipress.
NISEE. n.d. “The earthquake engineering online archive.” Accessed February 12, 2017. https://nisee.berkeley.edu/elibrary/.
Parkhi, O. M., A. Vedaldi, and A. Zisserman. 2015. “Deep face recognition.” In Vol. 1 of Proc., British Machine Vision Conf., 6. Surrey, UK: British Machine Vision Association Press.
PEER (Pacific Earthquake Engineering Research Center). n.d.-a. “Call for uploading images for PHI (PEER Hub ImageNet) Challenge.” Accessed January 15, 2018. https://apps.peer.berkeley.edu/spo.
PEER (Pacific Earthquake Engineering Research Center). n.d.-b. “Peer Hub ImageNet (-Net).” Accessed June 20, 2019. https://apps.peer.berkeley.edu/phi-net/.
PEER (Pacific Earthquake Engineering Research Center). n.d.-c. “ Challenge 2018.” Accessed January 15, 2018. https://apps.peer.berkeley.edu/phichallenge.
Santos, J. P., C. Cremona, A. D. Orcesi, and P. Silveira. 2017. “Early damage detection based on pattern recognition and data fusion.” J. Struct. Eng. 143 (2): 04016162. https://doi.org/10.1061/(ASCE)ST.1943-541X.0001643.
Sezen, H., A. S. Whittaker, K. J. Elwood, and K. M. Mosalam. 2003. “Performance of reinforced concrete buildings during the August 17, 1999 Kocaeli, Turkey earthquake, and seismic design and construction practice in Turkey.” Eng. Struct. 25 (1): 103–114. https://doi.org/10.1016/S0141-0296(02)00121-9.
Shi, Y., L. Cui, Z. Qi, F. Meng, and Z. Chen. 2016. “Automatic road crack detection using random structured forests.” IEEE Trans. Intell. Transp. Syst. 17 (12): 3434–3445. https://doi.org/10.1109/TITS.2016.2552248.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. https://arxiv.org/abs/1409.1556.
Srivastava, N., G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. 2014. “Dropout: A simple way to prevent neural networks from overfitting.” J. Mach. Learn. Res. 15 (1): 1929–1958.
Xu, Y., Y. Bao, J. Chen, W. Zuo, and H. Li. 2019. “Surface fatigue crack identification in steel box girder of bridges by a deep fusion convolutional neural network based on consumer-grade camera images.” Struct. Health Monit. 18 (3): 653–674. https://doi.org/10.1177/1475921718764873.
Xu, Y., S. Li, D. Zhang, Y. Jin, F. Zhang, N. Li, and H. Li. 2018. “Identification framework for cracks on a steel structure surface by a restricted Boltzmann machines algorithm based on consumer-grade camera images.” Struct. Control Health Monit. 25 (2): e2075. https://doi.org/10.1002/stc.2075.
Yang, Y., and S. Nagarajaiah. 2016. “Dynamic imaging: Real-time detection of local structural damage with blind separation of low-rank background and sparse innovation.” J. Struct. Eng. 142 (2): 04015144. https://doi.org/10.1061/(ASCE)ST.1943-541X.0001334.
Yeum, C. M., S. J. Dyke, and J. Ramirez. 2018. “Visual data classification in post-event building reconnaissance.” Eng. Struct. 155 (Jan): 16–24. https://doi.org/10.1016/j.engstruct.2017.10.057.
Yeum, C. M., A. Mohan, S. J. Dyke, M. Jahanshahi, J. Choi, Z. Zhao, A. Lenjani, and J. A. Ramirez. 2017. Image-based collection and measurements for construction pay items. West Lafayette, IN: Purdue Univ.
Zhang, A., K. C. Wang, B. Li, E. Yang, X. Dai, Y. Peng, Y. Fei, Y. Liu, J. Q. Li, and C. Chen. 2017. “Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network.” Comput.-Aided Civ. Infrastruct. Eng. 32 (10): 805–819. https://doi.org/10.1111/mice.12297.
Information & Authors
Information
Published In
Journal of Structural Engineering
Volume 146 • Issue 10 • October 2020
Copyright
© 2020 American Society of Civil Engineers.
History
Received: Sep 5, 2019
Accepted: Mar 16, 2020
Published online: Jul 17, 2020
Published in print: Oct 1, 2020
Discussion open until: Dec 17, 2020
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.