Style-Controlled Image Synthesis of Concrete Damages Based on Fusion of Convolutional Encoder and Attention-Enhanced Conditional Generative Adversarial Network
Publication: Journal of Computing in Civil Engineering
Volume 38, Issue 6
Abstract
Developing deep learning network models for computer vision applications in concrete damage detection is a challenging task due to the shortage of training images. To address this issue, this study proposes a novel style-controlled image synthesis method for concrete damages based on the fusion of a convolutional encoder and an attention-enhanced conditional generative adversarial network. This makes it possible to generate effective images that can improve the damage detection performance of deep learning networks. To achieve this, a network architecture for concrete damage image synthesis, named DamageGAN-AE, was designed by fusing a convolutional encoder and an attention-enhanced conditional generative adversarial network. The DamageGAN-AE networks with different attention modules were trained, and the training results show that the well-trained DamageGAN-AE enhanced by coordinate attention is the best model for concrete damage image synthesis. The well-trained DamageGAN-AE was compared with the current competing methods to verify its performance. The DamageGAN-AE with image encoder was trained to implement the style-controlled image synthesis. Finally, the generated concrete damage images with diverse styles by the DamageGAN-AE model with image encoder were used to train deep learning networks. The results indicate that the generated style-controlled concrete damage images by the proposed method can effectively improve the concrete damage detection performance of deep learning networks.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was supported by the National Natural Science Foundation of China (No. 52308333) and the China Postdoctoral Science Foundation (No. 2022M723401).
References
Abdel-Qader, I., O. Abudayyeh, and M. Kelly. 2003. “Analysis of edge detection techniques for crack identification in bridges.” J. Comput. Civ. Eng. 17 (4): 255–263. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(255).
Adhikari, R., O. Moselhi, and A. Bagchi. 2014. “Image-based retrieval of concrete crack properties for bridge inspection.” Autom. Constr. 39 (Dec): 180–194. https://doi.org/10.1016/j.autcon.2013.06.011.
Ali, L., F. Alnajjar, W. Khan, M. A. Serhani, and H. Al Jassmi. 2022. “Bibliometric analysis and review of deep learning-based crack detection literature published between 2010 and 2022.” Buildings 12 (4): 432. https://doi.org/10.3390/buildings12040432.
Baek, F., D. Kim, S. Park, H. Kim, and S. Lee. 2022. “Conditional generative adversarial networks with adversarial attack and defense for generative data augmentation.” J. Comput. Civ. Eng. 36 (3): 04022001. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001015.
Cha, Y. J., W. Choi, and O. Büyüköztürk. 2017. “Deep learning-based crack damage detection using convolutional neural networks.” Comput.-Aided Civ. Infrastruct. Eng. 32 (5): 361–378. https://doi.org/10.1111/mice.12263.
Cheng, H., X. J. Shi, and C. Glazier. 2003. “Real-time image thresholding based on sample space reduction and interpolation approach.” J. Comput. Civ. Eng. 17 (4): 264–272. https://doi.org/10.1061/(ASCE)0887-3801(2003)17:4(264).
Dunphy, K., A. Sadhu, and J. Wang. 2022. “Multiclass damage detection in concrete structures using a transfer learning-based generative adversarial networks.” Struct. Control Health 29 (Jun): e3079. https://doi.org/10.1002/stc.3079.
Ellingson, S. R., B. Davis, and J. Allen. 2020. “Machine learning and ligand binding predictions: A review of data, methods, and obstacles.” Biochim. Biophys. Acta, Gen. Subj. 1864 (6): 129545. https://doi.org/10.1016/j.bbagen.2020.129545.
Gao, Y., B. Kong, and K. M. Mosalam. 2019. “Deep leaf-bootstrapping generative adversarial network for structural image data augmentation.” Comput.-Aided Civ. Infrastruct. Eng. 34 (9): 755–773. https://doi.org/10.1111/mice.12458.
German, S., I. Brilakis, and R. DesRoches. 2012. “Rapid entropy-based detection and properties measurement of concrete spalling with machine vision for post-earthquake safety assessments.” Adv. Eng. Inf. 26 (4): 846–858. https://doi.org/10.1016/j.aei.2012.06.005.
Graybeal, B. A., B. M. Phares, D. D. Rolander, M. Moore, and G. Washer. 2002. “Visual inspection of highway bridges.” J. Nondestr. Eval. 21 (3): 67–83. https://doi.org/10.1023/A:1022508121821.
Hou, Q., D. Zhou, and J. Feng. 2021. “Coordinate attention for efficient mobile network design.” In Proc., 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 13708–13717. New York: IEEE.
Hou, X., K. Sun, L. Shen, and G. Qiu. 2019. “Improving variational autoencoder with deep feature consistent and generative adversarial training.” Neurocomputing 341 (Dec): 183–194. https://doi.org/10.1016/j.neucom.2019.03.013.
Hu, J., L. Shen, and G. Sun. 2018. “Squeeze-and-excitation networks.” In Proc., 2018 IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 7132–7141. New York: IEEE.
Isola, P., J. Y. Zhu, T. Zhou, and A. A. Efros. 2017. “Image-to-image translation with conditional adversarial networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 5967–5976. New York: IEEE.
Kirschke, K., and S. Velinsky. 1992. “Histogram-based approach for automated pavement-crack sensing.” J. Transp. Eng. 118 (5): 700–710. https://doi.org/10.1061/(ASCE)0733-947X(1992)118:5(700).
Li, S., and X. Zhao. 2019. “Image-based concrete crack detection using convolutional neural network and exhaustive search technique.” Adv. Civ. Eng. 2019 (Mar): 6520620. https://doi.org/10.1155/2019/6520620.
Li, S., and X. Zhao. 2021. “Pixel-level detection and measurement of concrete crack using faster region-based convolutional neural network and morphological feature extraction.” Meas. Sci. Technol. 32 (Dec): 065010. https://doi.org/10.1088/1361-6501/abb274.
Li, S., and X. Zhao. 2022. “A performance improvement strategy for concrete damage detection using stacking ensemble learning of multiple semantic segmentation networks.” Sensors 22 (9): 3341. https://doi.org/10.3390/s22093341.
Li, S., and X. Zhao. 2023. “High-resolution concrete damage image synthesis using conditional generative adversarial network.” Autom. Constr. 147 (Dec): 104739. https://doi.org/10.1016/j.autcon.2022.104739.
Li, S., X. Zhao, and G. Zhou. 2019. “Automatic pixel-level multiple damage detection of concrete structure using fully convolutional network.” Comput.-Aided Civ. Infrastruct. Eng. 34 (7): 616–634. https://doi.org/10.1111/mice.12433.
Long, J., E. Shelhamer, and T. Darrell. 2015. “Fully convolutional networks for semantic segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 3431–3440. New York: IEEE.
Maeda, H., T. Kashiyama, Y. Sekimoto, T. Seto, and H. Omata. 2021. “Generative adversarial network for road damage detection.” Comput.-Aided Civ. Infrastruct. Eng. 36 (1): 47–60. https://doi.org/10.1111/mice.12561.
Mao, X., Q. Li, H. Xie, Y. R. Lau, Z. Wang, and S. P. Smolley. 2017. “Least squares generative adversarial networks.” In Proc., IEEE Int. Conf. on Computer Vision (ICCV), 2813–2821. New York: IEEE.
Mei, Q., M. Gül, and M. R. Azim. 2020. “Densely connected deep neural network considering connectivity of pixels for automatic crack detection.” Autom. Constr. 110 (Feb): 103018. https://doi.org/10.1016/j.autcon.2019.103018.
Miyato, T., T. Kataoka, M. Koyama, and Y. Yoshida. 2018. “Spectral normalization for generative adversarial networks.” Preprint, submitted February 16, 2018. https://arxiv.org/abs/1802.05957.
Park, T., M.-Y. Liu, T.-C. Wang, and J.-Y. Zhu. 2019. “Semantic image synthesis with spatially-adaptive normalization.” In Proc., 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 2332–2341. New York: IEEE.
Pei, L., Z. Sun, L. Xiao, W. Li, J. Sun, and H. Zhang. 2021. “Virtual generation of pavement crack images based on improved deep convolutional generative adversarial network.” Eng. Appl. Artif. Intell. 104 (Dec): 104376. https://doi.org/10.1016/j.engappai.2021.104376.
Qu, Z., C.-Y. Wang, S.-Y. Wang, and F.-R. Ju. 2022. “A method of hierarchical feature fusion and connected attention architecture for pavement crack detection.” IEEE Trans. Intell. Transp. Syst. 23 (9): 16038–16047. https://doi.org/10.1109/TITS.2022.3147669.
Shin, H., Y. Ahn, S. Tae, H. Gil, M. Song, and S. Lee. 2021. “Enhancement of multi-class structural defect recognition using generative adversarial network.” Sustainability 13 (22): 12682. https://doi.org/10.3390/su132212682.
Shorten, C., and T. M. Khoshgoftaar. 2019. “A survey on image data augmentation for deep learning.” J. Big Data 6 (1): 1–48. https://doi.org/10.1186/s40537-019-0197-0.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. https://arxiv.org/abs/1409.1556.
Sushko, V., E. Schönfeld, D. Zhang, J. Gall, B. Schiele, and A. Khoreva. 2021. “You only need adversarial supervision for semantic image synthesis.” Preprint, submitted March 19, 2021. https://arxiv.org/abs/2012.04781.
Talab, A. M. A., Z. Huang, F. Xi, and L. Haiming. 2016. “Detection crack in image using Otsu method and multiple filtering in image processing techniques.” Optik 127 (3): 1030–1033. https://doi.org/10.1016/j.ijleo.2015.09.147.
Wang, M., and J. C. P. Cheng. 2020. “A unified convolutional neural network integrated with conditional random field for pipe defect segmentation.” Comput.-Aided Civ. Infrastruct. Eng. 35 (Dec): 162–177. https://doi.org/10.1111/mice.12481.
Wang, Q., B. Wu, P. Zhu, P. Li, W. Zuo, and Q. Hu. 2020. “ECA-Net: Efficient channel attention for deep convolutional neural networks.” In Proc., Conf. on Computer Vision and Pattern Recognition (CVPR), 11531–11539. New York: IEEE.
Wang, T. C., M. Y. Liu, J. Y. Zhu, A. Tao, J. Kautz, and B. Catanzaro. 2018. “High-resolution image synthesis and semantic manipulation with conditional GANs.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 8798–8807. New York: IEEE.
Yang, L., R. Zhang, L. Li, and X. Xie. 2021. “SimAM: A simple, parameter-free attention module for convolutional neural networks.” In Vol. 139 of Proc., Int. Conf. on Machine Learning, 11863–11874. Breckenridge, CO: Proceedings of Machine Learning Research.
Yang, X., H. Li, Y. Yu, X. Luo, T. Huang, and X. Yang. 2018. “Automatic pixel-level crack detection and measurement using fully convolutional network.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12): 1090–1109. https://doi.org/10.1111/mice.12412.
Yao, H., Y. Liu, X. Li, Z. You, Y. Feng, and W. Lu. 2022. “A detection method for pavement cracks combining object detection and attention mechanism.” IEEE Trans. Intell. Transp. Syst. 23 (11): 22179–22189. https://doi.org/10.1109/TITS.2022.3177210.
Yeum, C., and S. Dyke. 2015. “Vision-based automated crack detection for bridge inspection.” Comput.-Aided Civ. Infrastruct. Eng. 30 (Dec): 759–770. https://doi.org/10.1111/mice.12141.
Yu, S., J. Jang, and C. Han. 2007. “Auto inspection system using a mobile robot for detecting concrete cracks in a tunnel.” Autom. Constr. 16 (Mar): 255–261. https://doi.org/10.1016/j.autcon.2006.05.003.
Zhan, F., S. Lu, C. Zhang, F. Ma, and X. Xie. 2021. “Adversarial image composition with auxiliary illumination.” In Proc., 2020 Asian Conf. on Computer Vision, 234–250. New York: Springer.
Zhang, H., I. Goodfellow, D. Metaxas, and A. Odena. 2019. “Self-attention generative adversarial networks.” In Proc., 36th Int. Conf. on Machine Learning, 7354–7363. Long Beach, CA: Proceedings of Machine Learning Research.
Zhang, H., T. Xu, H. Li, S. Zhang, X. Wang, X. Huang, and D. N. Metaxas. 2018. “StackGAN++: Realistic image synthesis with stacked generative adversarial networks.” IEEE Trans. Pattern Anal. Mach. Intell. 41 (8): 1947–1962. https://doi.org/10.1109/TPAMI.2018.2856256.
Zhang, L., J. Shen, and B. Zhu. 2021. “A research on an improved Unet-based concrete crack detection algorithm.” Struct. Health Monit. 20 (4): 1864–1879. https://doi.org/10.1177/1475921720940068.
Zhao, P., W. Liao, Y. Huang, and X. Lu. 2023. “Intelligent design of shear wall layout based on attention-enhanced generative adversarial network.” Eng. Struct. 274 (Dec): 115170. https://doi.org/10.1016/j.engstruct.2022.115170.
Zhu, J., J. Zhong, T. Ma, X. Huang, W. Zhang, and Y. Zhou. 2021a. “Pavement distress detection using convolutional neural networks with images captured via UAV.” Autom. Constr. 133 (Jan): 103991. https://doi.org/10.1016/j.autcon.2021.103991.
Zhu, J. Y., T. Park, P. Isola, and A. A. Efros. 2017. “Unpaired image-to-image translation using cycle-consistent adversarial networks.” In Proc., IEEE Int. Conf. on Computer Vision, 2242–2251. New York: IEEE.
Zhu, X., S. Lyu, X. Wang, and Q. Zhao. 2021b. “TPH-YOLOv5: Improved YOLOv5 based on transformer prediction head for object detection on drone-captured scenarios.” Preprint, submitted April 23, 2021. https://arxiv.org/abs/2108.11539.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 38 • Issue 6 • November 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Feb 19, 2024
Accepted: May 21, 2024
Published online: Aug 6, 2024
Published in print: Nov 1, 2024
Discussion open until: Jan 6, 2025
ASCE Technical Topics:
- Artificial intelligence (AI)
- Artificial intelligence and machine learning
- Computer models
- Computer networks
- Computer programming
- Computer vision and image processing
- Computing in civil engineering
- Concrete
- Damage (material)
- Education
- Engineering fundamentals
- Engineering materials (by type)
- Materials characterization
- Materials engineering
- Methodology (by type)
- Models (by type)
- Neural networks
- Practice and Profession
- Training
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.