Structural Crack Detection from Benchmark Data Sets Using Pruned Fully Convolutional Networks
Publication: Journal of Structural Engineering
Volume 147, Issue 11
Abstract
Crack inspection is a crucial but labor-intensive work of maintenance for in-service bridges. Recently, the development of fully convolutional network (FCN) provides pixel-wise semantic segmentation, which is promising as a means of automatic crack detection. However, the demand for numerous training images with pixel-wise labels poses challenges. In this study, a benchmark data set called a bridge crack library (BCL) containing 11,000 pixel-wise labeled images with resolution was established, which has 5,769 nonsteel crack images, 2,036 steel crack images, 3,195 noise images, and their labels. It is aimed at crack detection on multiple structural materials including masonry, concrete, and steel. The raw images were collected by multiple cameras from more than 50 in-service bridges during a period of 2 years. Various crack images with numerous crack forms and noise motifs in different scenarios were collected. Quality control measures were carried out during the raw image collection, subimage cropping, and subimage annotation steps. The established BCL was used to train three deep neural networks (DNNs) for applicability validation. The results indicate that the BCL could be applied to effectively train DNNs for crack detection and serve as a benchmark data set for performance evaluation of DNN models.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are available in a repository or online in accordance with funder data retention policies. The image data set is stored in the Harvard dataverse and can be visited through the following link: https://doi.org/10.7910/DVN/RURXSH.
It is uploaded as a zip file and consists of three categories: nonsteel crack images, steel crack images, and noise images. In each of the categories, there are two folders, one contains the color images and the other contains the corresponding labels. Users can download the file and decompress the zip file to get the images and labels.
Acknowledgments
The work described in this paper was jointly supported by the Basic Science Center Program for Multiphase Evolution in Hypergravity of the National Natural Science Foundation of China (Grant No. 51988101), the National Natural Science Foundation of China (Grant Nos. 51822810 and 51778574), and the Zhejiang Provincial Natural Science Foundation of China (Grant No. LR19E080002). The authors thank Professor Hui Li of the Harbin Institute of Technology, China for providing part of the crack image data used in this study during the First International Project Competition for Structural Health Monitoring (IPC-SHM 2020).
References
Alipour, M., D. K. Harris, and G. R. Miller. 2019. “Robust pixel-level crack detection using deep fully convolutional neural networks.” J. Comput. Civ. Eng. 33 (6): 04019040. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000854.
Bang, S., S. Park, H. Kim, and H. Kim. 2019. “Encoder-decoder network for pixel-level road crack detection in black-box images.” Comput.-Aided Civ. Infrastruct. Eng. 34 (8): 713–727. https://doi.org/10.1111/mice.12440.
Cha, Y. J., W. Choi, and O. Buyukozturk. 2017. “Deep learning-based crack damage detection using convolutional neural networks.” Comput.-Aided Civ. Infrastruct. Eng. 32 (5): 361–378. https://doi.org/10.1111/mice.12263.
Cha, Y. J., W. Choi, G. Suh, S. Mahmoudkhani, and O. Buyukozturk. 2018. “Autonomous structural visual inspection using region-based deep learning for detecting multiple damage types.” Comput.-Aided Civ. Infrastruct. Eng. 33 (9): 731–747. https://doi.org/10.1111/mice.12334.
Chen, L. C., Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. 2018. “Encoder-decoder with Atrous separable convolution for semantic image segmentation.” In Proc., European Conf. on Computer Vision. Berlin: Springer.
Chen, T. Y., Z. H. Cai, X. Zhao, C. Chen, X. F. Lianga, T. R. Zou, and P. Wang. 2020. “Pavement crack detection and recognition using the architecture of segNet.” J. Ind. Inf. Integr. 18 (Jun): 100144. https://doi.org/10.1016/j.jii.2020.100144.
Daneshgaran, F., L. Zacheo, F. Di Stasio, and M. Mondin. 2019. “Use of deep learning for automatic detection of cracks in tunnels: Prototype-2 developed in the 2017-2018 time period.” Transp. Res. Rec. 2673 (9): 44–50. https://doi.org/10.1177/0361198119845656.
Deng, J. H., Y. Lu, and V. C. S. Lee. 2021. “Imaging-based crack detection on concrete surfaces using you only look once network.” Struct. Health Monit. 20 (2): 484–499. https://doi.org/10.1177/1475921720938486.
Dorafshan, S., and H. Azari. 2020. “Deep learning models for bridge deck evaluation using impact echo.” Constr. Build. Mater. 263 (Dec): 120109. https://doi.org/10.1016/j.conbuildmat.2020.120109.
Dorafshan, S., R. J. Thomas, and M. Maguire. 2018a. “Comparison of deep convolutional neural networks and edge detectors for image-based crack detection in concrete.” Constr. Build. Mater. 186 (Oct): 1031–1045. https://doi.org/10.1016/j.conbuildmat.2018.08.011.
Dorafshan, S., R. J. Thomas, and M. Maguire. 2018b. “SDNET2018: An annotated image dataset for non-contact concrete crack detection using deep convolutional neural networks.” Data Brief 21 (Dec): 1664–1668. https://doi.org/10.1016/j.dib.2018.11.015.
Du, Y. C., N. Pan, Z. H. Xu, F. W. Deng, Y. Shen, and H. Kang. 2020. “Pavement distress detection and classification based on YOLO network.” Int. J. Pavement Eng. 1–14. https://doi.org/10.1080/10298436.2020.1714047.
Dung, C. V., and L. D. Anh. 2019. “Autonomous concrete crack detection using deep fully convolutional neural network.” Autom. Constr. 99 (Mar): 52–58. https://doi.org/10.1016/j.autcon.2018.11.028.
Feng, C. C., H. Zhang, S. Wang, Y. L. Li, H. R. Wang, and F. Yan. 2019. “Structural damage detection using deep convolutional neural network and transfer learning.” KSCE J. Civ. Eng. 23 (10): 4493–4502. https://doi.org/10.1007/s12205-019-0437-z.
Gao, X. W., M. Jian, M. Hu, M. Tanniru, and S. Q. Li. 2019. “Faster multi-defect detection system in shield tunnel using combination of FCN and Faster RCNN.” Adv. Struct. Eng. 22 (13): 2907–2921. https://doi.org/10.1177/1369433219849829.
Gao, Y. Q., and K. M. Mosalam. 2020. “PEER Hub ImageNet: A large-scale multiattribute benchmark data set of structural images.” J. Struct. Eng. 146 (10): 04020198. https://doi.org/10.1061/(ASCE)ST.1943-541X.0002745.
Gopalakrishnan, K., S. K. Khaitan, A. Choudhary, and A. Agrawal. 2017. “Deep convolutional neural networks with transfer learning for computer vision-based data-driven pavement distress detection.” Constr. Build. Mater. 157 (Dec): 322–330. https://doi.org/10.1016/j.conbuildmat.2017.09.110.
Hinton, G. E., and R. R. Salakhutdinov. 2006. “Reducing the dimensionality of data with neural networks.” Science 313 (5786): 504–507. https://doi.org/10.1126/science.1127647.
Hoang, N. D., Q. L. Nguyen, and V. D. Tran. 2018. “Automatic recognition of asphalt pavement cracks using metaheuristic optimized edge detection algorithms and convolution neural network.” Autom. Constr. 94 (Oct): 203–213. https://doi.org/10.1016/j.autcon.2018.07.008.
Huang, H. W., Q. T. Li, and D. M. Zhang. 2018. “Deep learning based image recognition for crack and leakage defects of metro shield tunnel.” Tunnelling Underground Space Technol. 77 (Jul): 166–176. https://doi.org/10.1016/j.tust.2018.04.002.
Ju, H. Y., W. Li, S. S. Tighe, Z. C. Xu, and J. Z. Zhai. 2020. “CrackU-net: A novel deep convolutional neural network for pixelwise pavement crack detection.” Struct. Control Health Monit. 27 (8): e2551. https://doi.org/10.1002/stc.2551.
Kang, D. H., and Y. J. Cha. 2018. “Autonomous UAVs for structural health monitoring using deep learning and an ultrasonic beacon system with geo-tagging.” Comput.-Aided Civ. Infrastruct. Eng. 33 (10): 885–902. https://doi.org/10.1111/mice.12375.
Kim, B., and S. Cho. 2018. “Automated vision-based detection of cracks on concrete surfaces using a deep learning technique.” Sensors 18 (10): 3452. https://doi.org/10.3390/s18103452.
Kim, B., and S. Cho. 2019. “Image-based concrete crack assessment using mask and region-based convolutional neural network.” Struct. Control Health Monit. 26 (8): e2381. https://doi.org/10.1002/stc.2381.
Kingma, D. P., and J. L. Ba. 2015. “Adam: A method for stochastic optimization.” Preprint, submitted December 22, 2014. https://arxiv.org/abs/1412.6980.
LeCun, Y., Y. Bengio, and G. Hinton. 2015. “Deep learning.” Nature 521 (7553): 436–444. https://doi.org/10.1038/nature14539.
Li, C., P. J. Xu, L. J. L. Niu, Y. Chen, L. S. Sheng, and M. C. Liu. 2019. “Tunnel crack detection using coarse-to-fine region localization and edge detection.” Wiley Interdiscip. Rev.: Data Min. Knowl. Discovery 9 (5): e1308. https://doi.org/10.1002/widm.1308.
Li, G., B. Ma, S. H. He, X. L. Ren, and Q. W. Liu. 2020. “Automatic tunnel crack detection based on U-net and a convolutional neural network with alternately updated clique.” Sensors 20 (3): 717. https://doi.org/10.3390/s20030717.
Liu, Q. 2019. “U-Net implementation in PyTorch.” Accessed December 12, 2019. https://github.com/Qiuyan918/Unet_Implementation_PyTorch/blob/master/Unet_Implementation_PyTorch.ipynb.
Liu, Z., J. Li, Z. Shen, G. Huang, S. Yan, and C. Zhang. 2017. “Learning efficient convolutional networks through network slimming.” In Proc., 2017 IEEE Int. Conf. on Computer Vision. New York: IEEE.
Long, J., E. Shelhamer, and T. Darrell. 2015. “Fully convolutional networks for semantic segmentation.” In Proc., 2015 IEEE Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Maeda, H., Y. Sekimoto, T. Seto, T. Kashiyama, and H. Omata. 2018. “Road damage detection and classification using deep neural networks with smartphone images.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12): 1127–1141. https://doi.org/10.1111/mice.12387.
Majidifard, H., P. Jin, Y. Adu-Gyamfi, and W. G. Buttlar. 2020. “Pavement image datasets: A new benchmark dataset to classify and densify pavement distresses.” Transp. Res. Rec. 2674 (2): 328–339. https://doi.org/10.1177/0361198120907283.
Mei, Q. P., and M. Gul. 2020. “A cost effective solution for pavement crack inspection using cameras and deep neural networks.” Constr. Build. Mater. 256 (Sep): 119397. https://doi.org/10.1016/j.conbuildmat.2020.119397.
Ni, F. T., J. Zhang, and Z. Q. Chen. 2019. “Pixel-level crack delineation in images with convolutional feature fusion.” Struct. Control Health Monit. 26 (1): e2286. https://doi.org/10.1002/stc.2286.
Ni, Y. Q., X. W. Ye, and J. M. Ko. 2010. “Monitoring-based fatigue reliability assessment of steel bridges: Analytical model and application.” J. Struct. Eng. 136 (12): 1563–1573. https://doi.org/10.1061/(ASCE)ST.1943-541X.0000250.
Ni, Y. Q., X. W. Ye, and J. M. Ko. 2012. “Modeling of stress spectrum using long-term monitoring data and finite mixture distributions.” J. Eng. Mech. 138 (2): 175–183. https://doi.org/10.1061/(ASCE)EM.1943-7889.0000313.
Ochoa-Ruiz, G., A. A. Angulo-Murillo, A. Ochoa-Zezzatti, L. M. Aguilar-Lobo, J. A. Vega-Fernandez, and S. Natraj. 2020. “An asphalt damage dataset and detection system based on RetinaNet for road conditions assessment.” Appl. Sci. 10 (11): 3974. https://doi.org/10.3390/app10113974.
Park, S., S. Bang, H. Kim, and H. Kim. 2019. “Patch-based crack detection in black box images using convolutional neural networks.” J. Comput. Civ. Eng. 33 (3): 04019017. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000831.
Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-Net: Convolutional networks for biomedical image segmentation.” In Proc., 18th Int. Conf. on Medical Image Computing and Computer Assisted Intervention. Berlin: Springer.
Ryu, E., J. Kang, J. Lee, Y. Shin, and H. Kim. 2020. “Automated detection of surface cracks and numerical correlation with thermal-structural behaviors of fire damaged concrete beams.” Int. J. Concr. Struct. Mater. 14 (1): 1–12. https://doi.org/10.1186/s40069-019-0387-3.
Sajedi, S. O., and X. Liang. 2019. “A convolutional cost-sensitive crack localization algorithm for automated and reliable RC bridge inspection.” In Proc., 10th New York City Bridge Conf. Boca Raton, FL: CRC Press.
Song, Q., Y. Q. Wu, X. S. Xin, L. Yang, M. Yang, H. M. Chen, C. Liu, M. J. Hu, X. S. Chai, and J. C. Li. 2019. “Real-time tunnel crack analysis system via deep learning.” IEEE Access 7 (May): 64186–64197. https://doi.org/10.1109/ACCESS.2019.2916330.
Song, W. D., G. H. Jia, H. Zhu, D. Jia, and L. Gao. 2020. “Automated pavement crack damage detection using deep multiscale convolutional features.” J. Adv. Transp. 2020: 6412562. https://doi.org/10.1155/2020/6412562.
Spencer, B. F., V. Hoskere, and Y. Narazaki. 2019. “Advances in computer vision-based civil infrastructure inspection and monitoring.” Engineering 5 (2): 199–222. https://doi.org/10.1016/j.eng.2018.11.030.
Tong, Z., J. Gao, Z. Q. Han, and Z. J. Wang. 2018. “Recognition of asphalt pavement crack length using deep convolutional neural networks.” Road Mater. Pavement Des. 19 (6): 1334–1349. https://doi.org/10.1080/14680629.2017.1308265.
Tong, Z., J. Gao, and H. T. Zhang. 2017. “Recognition, location, measurement, and 3D reconstruction of concealed cracks using convolutional neural networks.” Constr. Build. Mater. 146 (Aug): 775–787. https://doi.org/10.1016/j.conbuildmat.2017.04.097.
Wu, R. T., A. Singla, M. R. Jahanshahi, E. Bertino, B. J. Ko, and D. Verma. 2019. “Pruning deep convolutional neural networks for efficient edge computing in condition assessment of infrastructures.” Comput.-Aided Civ. Infrastruct. Eng. 34 (9): 774–789. https://doi.org/10.1111/mice.12449.
Xu, Y., S. Y. Wei, Y. Q. Bao, and H. Li. 2019. “Automatic seismic damage identification of reinforced concrete columns from images by a region-based deep convolutional neural network.” Struct. Control Health Monit. 26 (3): e2313. https://doi.org/10.1002/stc.2313.
Xue, Y. D., and Y. C. Li. 2018. “A fast detection method via region-based fully convolutional neural networks for shield tunnel lining defects.” Comput.-Aided Civ. Infrastruct. Eng. 33 (8): 638–654. https://doi.org/10.1111/mice.12367.
Yang, X. C., H. Li, Y. T. Yu, X. C. Luo, T. Huang, and X. Yang. 2018. “Automatic pixel-level crack detection and measurement using fully convolutional network.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12): 1090–1109. https://doi.org/10.1111/mice.12412.
Ye, X. W., T. Jin, and P. Y. Chen. 2019a. “Structural crack detection using deep learning-based fully convolutional networks.” Adv. Struct. Eng. 22 (16): 3412–3419. https://doi.org/10.1177/1369433219836292.
Ye, X. W., T. Jin, and C. B. Yun. 2019b. “A review on deep learning-based structural health monitoring of civil infrastructures.” Smart Struct. Syst. 24 (5): 567–585. https://doi.org/10.12989/sss.2019.24.5.567.
Ye, X. W., Y. Q. Ni, T. T. Wai, K. Y. Wong, X. M. Zhang, and F. Xu. 2013. “A vision-based system for dynamic displacement measurement of long-span bridges: Algorithm and verification.” Smart Struct. Syst. 12 (3–4): 363–379. https://doi.org/10.12989/sss.2013.12.3_4.363.
Ye, X. W., Y. Q. Ni, K. Y. Wong, and J. M. Ko. 2012. “Statistical analysis of stress spectra for fatigue life assessment of steel bridges with structural health monitoring data.” Eng. Struct. 45 (Dec): 166–176. https://doi.org/10.1016/j.engstruct.2012.06.016.
Yeum, C. M., J. Choi, and S. J. Dyke. 2019. “Automated region-of-interest localization and classification for vision-based visual assessment of civil infrastructure.” Struct. Health Monit. 18 (3): 675–689. https://doi.org/10.1177/1475921718765419.
Zhang, A., K. C. P. Wang, Y. Fei, Y. Liu, C. Chen, G. W. Yang, J. Q. Li, E. H. Yang, and S. Qiu. 2019a. “Automated pixel-level pavement crack detection on 3D asphalt surfaces with a recurrent neural network.” Comput.-Aided Civ. Infrastruct. Eng. 34 (3): 213–229. https://doi.org/10.1111/mice.12409.
Zhang, A., K. C. P. Wang, Y. Fei, Y. Liu, S. Y. Tao, C. Chen, J. Q. Li, and B. X. Li. 2018a. “Deep learning-based fully automated pavement crack detection on 3D asphalt surfaces with an improved CrackNet.” J. Comput. Civ. Eng. 32 (5): 04018041. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000775.
Zhang, C. B., C. C. Chang, and M. Jamshidi. 2020a. “Concrete bridge surface damage detection using a single-stage detector.” Comput.-Aided Civ. Infrastruct. Eng. 35 (4): 389–409. https://doi.org/10.1111/mice.12500.
Zhang, K. G., H. D. Cheng, and B. Y. Zhang. 2018b. “Unified approach to pavement crack and sealed crack detection using preclassification based on transfer learning.” J. Comput. Civ. Eng. 32 (2): 04018001. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000736.
Zhang, L. X., J. K. Shen, and B. J. Zhu. 2020b. “A research on an improved Unet-based concrete crack detection algorithm.” Struct. Health Monit. 20 (4): 1864–1879. https://doi.org/10.1177/1475921720940068.
Zhang, X. X., D. Rajan, and B. Story. 2019b. “Concrete crack detection using context-aware deep semantic segmentation network.” Comput.-Aided Civ. Infrastruct. Eng. 34 (11): 951–971. https://doi.org/10.1111/mice.12477.
Information & Authors
Information
Published In
Journal of Structural Engineering
Volume 147 • Issue 11 • November 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Nov 16, 2020
Accepted: Jun 2, 2021
Published online: Aug 27, 2021
Published in print: Nov 1, 2021
Discussion open until: Jan 27, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Mohammad Sadegh Barkhordari, Danial Jahed Armaghani, Panagiotis G. Asteris, Structural Damage Identification Using Ensemble Deep Convolutional Neural Network Models, Computer Modeling in Engineering & Sciences, 10.32604/cmes.2022.020840, 134, 2, (835-855), (2023).
- Junjie Wang, Ying Lei, Xiongjun Yang, Fubo Zhang, A refinement network embedded with attention mechanism for computer vision based post-earthquake inspections of railway viaduct, Engineering Structures, 10.1016/j.engstruct.2022.115572, 279, (115572), (2023).
- Xiao-Wei Ye, Zhen Sun, Jun Lu, Prediction and early warning of wind-induced girder and tower vibration in cable-stayed bridges with machine learning-based approach, Engineering Structures, 10.1016/j.engstruct.2022.115261, 275, (115261), (2023).
- Eric L Bianchi, Nazmus Sakib, Craig Woolsey, Matthew Hebdon, Bridge inspection component registration for damage evolution, Structural Health Monitoring, 10.1177/14759217221083647, 22, 1, (472-495), (2022).
- Younes Hamishebahar, Zahra Jadidi, Jun Jo, Crack segmentation enhancement using an adaptive hyper-parameter tuning scheme configured by a novel crack type clustering method, 2022 International Conference on Electrical, Computer and Energy Technologies (ICECET), 10.1109/ICECET55527.2022.9873023, (1-6), (2022).
- Kangcheng Liu, Guidong Yang, Jihan Zhang, Zuoquan Zhao, Xi Chen, Ben M. Chen, Datasets and Methods for Boosting Infrastructure Inspection: A Survey on Defect Segmentation and Detection, 2022 IEEE 17th International Conference on Control & Automation (ICCA), 10.1109/ICCA54724.2022.9831925, (23-30), (2022).
- Huu-Tai Thai, Machine learning for structural engineering: A state-of-the-art review, Structures, 10.1016/j.istruc.2022.02.003, 38, (448-491), (2022).
- Guidong Yang, Kangcheng Liu, Jihan Zhang, Benyun Zhao, Zuoquan Zhao, Xi Chen, Ben M. Chen, Datasets and processing methods for boosting visual inspection of civil infrastructure: A comprehensive review and algorithm comparison for crack classification, segmentation, and detection, Construction and Building Materials, 10.1016/j.conbuildmat.2022.129226, 356, (129226), (2022).
- Eric Bianchi, Matthew Hebdon, Visual structural inspection datasets, Automation in Construction, 10.1016/j.autcon.2022.104299, 139, (104299), (2022).