Conditional Generative Adversarial Networks with Adversarial Attack and Defense for Generative Data Augmentation
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 3
Abstract
Developing deep neural network (DNN) models for computer vision applications for construction is challenging due to the shortage of training data. To address this issue, we proposed a novel data augmentation method that integrates a conditional generative adversarial networks (GANs) framework with a target classifier. The integrated architecture enables adversarial attack and defense during end-to-end training, thereby making it possible to generate effective images for the target classifier’s training. We trained and tested two image classification DNNs with and without data augmentation, where we confirmed the effectiveness of the proposed method: with the data augmentation, the classification accuracy improved by 4.2 percentage points, from 71.24% to 75.46%, with qualitatively improved feature extraction more focused on the target object. Given that the application areas of our method are open-ended, the result is noteworthy. The proposed method can help construction researchers offset the data insufficiency, which will contribute to having more accurate and scalable DNN-powered vision models in construction applications.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was supported by grants from the National Research Foundation of Korea (NRF) funded by the Ministry of Science and ICT (No. 2018R1A2B2008600) and the Ministry of Education (No. 2018R1A6A1A08025348). This work was supported financially by a National Science Foundation Award (No. IIS-1734266, “Scene Understanding and Predictive Monitoring for Safe Human-Robot Collaboration in Unstructured and Dynamic Construction Environments”).
References
Alipour, M., D. K. Harris, and G. R. Miller. 2019. “Robust pixel-level crack detection using deep fully convolutional neural networks.” J. Comput. Civ. Eng. 33 (6): 04019040. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000854.
Alom, M. Z., T. M. Taha, C. Yakopcic, S. Westberg, P. Sidike, M. S. Nasrin, M. Hasan, B. C. Van Essen, A. A. Awwal, and V. K. Asari. 2019. “A state-of-the-art survey on deep learning theory and architectures.” Electronics 8 (3): 292. https://doi.org/10.3390/electronics8030292.
Arjovsky, M., and L. Bottou. 2017. “Towards principled methods for training generative adversarial networks.” Preprint, submitted January 17, 2017. http://arxiv.org/abs/1701.04862.
Bang, S., F. Baek, S. Park, W. Kim, and H. Kim. 2020. “Image augmentation to improve construction resource detection using generative adversarial networks, cut-and-paste, and image transformation techniques.” Autom. Constr. 115 (Jul): 103198. https://doi.org/10.1016/j.autcon.2020.103198.
Bang, S., S. Park, H. Kim, and H. Kim. 2019. “Encoder–decoder network for pixel-level road crack detection in black-box images.” Comput.-Aided Civ. Infrastruct. Eng. 34 (8): 713–727. https://doi.org/10.1111/mice.12440.
Barz, B., and J. Denzler. 2020. “Deep learning on small datasets without pre-training using cosine loss.” In Proc., IEEE/CVF Winter Conf. on Applications of Computer Vision, 1371–1380. New York: IEEE.
Birodkar, V., H. Mobahi, and S. Bengio. 2019. “Semantic redundancies in image-classification datasets: The 10% you don’t need.” Preprint, submitted January 29, 2019. http://arxiv.org/abs/1901.11409.
Bowles, C., L. Chen, R. Guerrero, P. Bentley, R. Gunn, A. Hammers, D. A. Dickie, M. V. Hernández, J. Wardlaw, and D. Rueckert. 2018. “GAN augmentation: Augmenting training data using generative adversarial networks.” Preprint, submitted October 25, 2018. http://arxiv.org/abs/1810.10863.
Braun, A., S. Tuttas, A. Borrmann, and U. Stilla. 2020. “Improving progress monitoring by fusing point clouds, semantic data and computer vision.” Autom. Constr. 116 (Aug): 103210. https://doi.org/10.1016/j.autcon.2020.103210.
Brock, A., J. Donahue, and K. Simonyan. 2018. “Large scale GAN training for high fidelity natural image synthesis.” Preprint, submitted September 28, 2018. http://arxiv.org/abs/1809.11096.
Cubuk, E. D., B. Zoph, D. Mane, V. Vasudevan, and Q. V. Le. 2019. “Autoaugment: Learning augmentation strategies from data.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 113–123. New York: IEEE.
Cubuk, E. D., B. Zoph, J. Shlens, and Q. V. Le. 2020. “Randaugment: Practical automated data augmentation with a reduced search space.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition Workshops, 702–703. New York: IEEE.
Davis, P., F. Aziz, M. T. Newaz, W. Sher, and L. Simon. 2021. “The classification of construction waste material using a deep convolutional neural network.” Autom. Constr. 122 (Feb): 103481. https://doi.org/10.1016/j.autcon.2020.103481.
Dung, C. V., H. Sekiya, S. Hirano, T. Okatani, and C. Miki. 2019. “A vision-based method for crack detection in gusset plate welded joints of steel bridges using deep convolutional neural networks.” Autom. Constr. 102 (Jun): 217–229. https://doi.org/10.1016/j.autcon.2019.02.013.
Duong, H.-T., and V. T. Hoang. 2019. “Data augmentation based on color features for limited training texture classification.” In Proc., 2019 4th Int. Conf. on Information Technology (InCIT), 208–211. New York: IEEE.
Dwibedi, D., I. Misra, and M. Hebert. 2017. “Cut, paste and learn: Surprisingly easy synthesis for instance detection.” In Proc., IEEE Int. Conf. on Computer Vision, 1301–1310. New York: IEEE.
Fang, Q., H. Li, X. Luo, L. Ding, H. Luo, T. M. Rose, and W. An. 2018a. “Detecting non-hardhat-use by a deep learning method from far-field surveillance videos.” Autom. Constr. 85 (Jan): 1–9. https://doi.org/10.1016/j.autcon.2017.09.018.
Fang, W., L. Ding, H. Luo, and P. E. Love. 2018b. “Falls from heights: A computer vision-based approach for safety harness detection.” Autom. Constr. 91 (Jul): 53–61. https://doi.org/10.1016/j.autcon.2018.02.018.
Fang, W., P. E. Love, H. Luo, and L. Ding. 2020. “Computer vision for behaviour-based safety in construction: A review and future directions.” Adv. Eng. Inf. 43 (Jan): 100980. https://doi.org/10.1016/j.aei.2019.100980.
Gil, D., G. Lee, and K. Jeon. 2018. “Classification of images from construction sites using a deep-learning algorithm.” In Proc., Int. Symp. on Automation and Robotics in Construction (ISARC), 1–6. Oulu, Finland: IAARC Publications.
Goodfellow, I. J., J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-Farley, S. Ozair, A. Courville, and Y. Bengio. 2014a. “Generative adversarial networks.” Preprint, submitted October 25, 2018. http://arxiv.org/abs/1406.2661.
Goodfellow, I. J., J. Shlens, and C. Szegedy. 2014b. “Explaining and harnessing adversarial examples.” Preprint, submitted December 20, 2014. http://arxiv.org/abs/1412.6572.
Grosse, K., T. Lee, Y. Park, M. Backes, and I. Molloy. 2020. “A new measure for overfitting and its implications for backdooring of deep learning.” Preprint, submitted June 11, 2020. http://arxiv.org/abs/2006.06721.
He, K., G. Gkioxari, P. Dollár, and R. Girshick. 2017. “Mask R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 2961–2969. New York: IEEE.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Huang, J., V. Rathod, C. Sun, M. Zhu, A. Korattikara, A. Fathi, I. Fischer, Z. Wojna, Y. Song, and S. Guadarrama. 2017. “Speed/accuracy trade-offs for modern convolutional object detectors.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 7310–7311. New York: IEEE.
Inoue, H. 2018. “Data augmentation by pairing samples for images classification.” Preprint, submitted January 9, 2018. http://arxiv.org/abs/1801.02929.
Isola, P., J.-Y. Zhu, T. Zhou, and A. A. Efros. 2017. “Image-to-image translation with conditional adversarial networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1125–1134. New York: IEEE.
Jang, Y., Y. Ahn, and H. Y. Kim. 2019a. “Estimating compressive strength of concrete using deep convolutional neural networks with digital microscope images.” J. Comput. Civ. Eng. 33 (3): 04019018. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000837.
Jang, Y., T. Zhao, S. Hong, and H. Lee. 2019b. “Adversarial defense via learning to generate diverse attacks.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 2740–2749. New York: IEEE.
Jiang, Y., D. Pang, and C. Li. 2021. “A deep learning approach for fast detection and classification of concrete damage.” Autom. Constr. 128 (Aug): 103785. https://doi.org/10.1016/j.autcon.2021.103785.
Jiao, L., F. Zhang, F. Liu, S. Yang, L. Li, Z. Feng, and R. Qu. 2019. “A survey of deep learning-based object detection.” IEEE Access 7: 128837–128868. https://doi.org/10.1109/ACCESS.2019.2939201.
Karras, T., T. Aila, S. Laine, and J. Lehtinen. 2017. “Progressive growing of GANS for improved quality, stability, and variation.” Preprint, submitted October 27, 2017. http://arxiv.org/abs/1710.10196.
Kim, D., M. Liu, S. Lee, and V. R. Kamat. 2019. “Remote proximity monitoring between mobile construction resources using camera-mounted UAVs.” Autom. Constr. 99 (Mar): 168–182. https://doi.org/10.1016/j.autcon.2018.12.014.
Kim, H., and H. Kim. 2018. “3D reconstruction of a concrete mixer truck for training object detectors.” Autom. Constr. 88 (Apr): 23–30. https://doi.org/10.1016/j.autcon.2017.12.034.
Kim, H., H. Kim, Y. W. Hong, and H. Byun. 2018. “Detecting construction equipment using a region-based fully convolutional network and transfer learning.” J. Comput. Civ. Eng. 32 (2): 04017082. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000731.
Kim, J., and S. Chi. 2020. “Multi-camera vision-based productivity monitoring of earthmoving operations.” Autom. Constr. 112 (Apr): 103121. https://doi.org/10.1016/j.autcon.2020.103121.
Kingma, D. P., and J. Ba. 2014. “Adam: A method for stochastic optimization.” Preprint, submitted December 22, 2014. http://arxiv.org/abs/1412.6980.
Laidlaw, C., and S. Feizi. 2019. “Functional adversarial attacks.” Preprint, submitted May 29, 2019. http://arxiv.org/abs/1906.00001.
Lei, L., Y. Zhou, H. Luo, and P. E. Love. 2019. “A CNN-based 3D patch registration approach for integrating sequential models in support of progress monitoring.” Adv. Eng. Inf. 41 (Aug): 100923. https://doi.org/10.1016/j.aei.2019.100923.
Liang, C.-J., K. M. Lundeen, W. McGee, C. C. Menassa, S. Lee, and V. R. Kamat. 2019. “A vision-based marker-less pose estimation system for articulated construction robots.” Autom. Constr. 104 (Aug): 80–94. https://doi.org/10.1016/j.autcon.2019.04.004.
Lin, T.-Y., M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. 2014. “Microsoft coco: Common objects in context.” In Proc., European Conf. on Computer Vision, 740–755. Cham, Switzerland: Springer.
Luo, H., M. Wang, P. K.-Y. Wong, and J. C. Cheng. 2020. “Full body pose estimation of construction equipment using computer vision and deep learning techniques.” Autom. Constr. 110 (Feb): 103016. https://doi.org/10.1016/j.autcon.2019.103016.
Luo, Z., S. Cheng, and Q. Zheng. 2019. “GAN-based augmentation for improving CNN performance of classification of defective photovoltaic module cells in electroluminescence images.” In Proc., IOP Conf. Series: Earth and Environmental Science, 012106. Bristol, UK: IOP Publishing.
Madry, A., A. Makelov, L. Schmidt, D. Tsipras, and A. Vladu. 2017. “Towards deep learning models resistant to adversarial attacks.” Preprint, submitted June 19, 2017. http://arxiv.org/abs/1706.06083.
Mikołajczyk, A., and M. Grochowski. 2018. “Data augmentation for improving deep learning in image classification problem.” In Proc., 2018 Int. Interdisciplinary PhD Workshop (IIPhDW), 117–122. New York: IEEE.
Milz, S., T. Rudiger, and S. Suss. 2018. “Aerial GANeration: Towards realistic data augmentation using conditional GANs.” In Proc., European Conf. on Computer Vision (ECCV) Workshops. Cham, Switzerland: Springer.
Mirza, M., and S. Osindero. 2014. “Conditional generative adversarial nets.” Preprint, submitted November 6, 2014. http://arxiv.org/abs/1411.1784.
Moosavi-Dezfooli, S.-M., A. Fawzi, O. Fawzi, and P. Frossard. 2017. “Universal adversarial perturbations.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1765–1773. New York: IEEE.
Moosavi-Dezfooli, S.-M., A. Fawzi, and P. Frossard. 2016. “Deepfool: A simple and accurate method to fool deep neural networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 2574–2582. New York: IEEE.
Moreno-Barea, F. J., J. M. Jerez, and L. Franco. 2020. “Improving classification accuracy using data augmentation on small data sets.” Expert Syst. Appl. 161 (Dec): 113696. https://doi.org/10.1016/j.eswa.2020.113696.
Papernot, N., P. McDaniel, X. Wu, S. Jha, and A. Swami. 2016. “Distillation as a defense to adversarial perturbations against deep neural networks.” In Proc., 2016 IEEE Symp. on Security and Privacy (SP), 582–597. New York: IEEE.
Park, S., S. Bang, H. Kim, and H. Kim. 2019. “Patch-based crack detection in black box images using convolutional neural networks.” J. Comput. Civ. Eng. 33 (3): 04019017. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000831.
Pei, K., Y. Cao, J. Yang, and S. Jana. 2017. “Deepxplore: Automated whitebox testing of deep learning systems.” In Proc., 26th Symp. on Operating Systems Principles, 1–18. New York: Association for Computing Machinery.
Perez, L., and J. Wang. 2017. “The effectiveness of data augmentation in image classification using deep learning.” Preprint, submitted December 13, 2017. http://arxiv.org/abs/1712.04621.
Radford, A., L. Metz, and S. Chintala. 2015. “Unsupervised representation learning with deep convolutional generative adversarial networks.” Preprint, submitted November 19, 2015. http://arxiv.org/abs/1511.06434.
Rao, J., and J. Zhang. 2017. “Cut and paste: Generate artificial labels for object detection.” In Proc., Int. Conf. on Video and Image Processing, 29–33. New York: Association for Computing Machinery.
Raoofi, H., and A. Motamedi. 2020. “Mask R-CNN deep learning-based approach to detect construction machinery on jobsites.” In Proc., Int. Symp. on Automation and Robotics in Construction (ISARC), 1122–1127. Amsterdam, Netherlands: IAARC Publications.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” Preprint, submitted June 4, 2015. http://arxiv.org/abs/1506.01497.
Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-net: Convolutional networks for biomedical image segmentation.” In Proc., Int. Conf. on Medical Image Computing and Computer-Assisted Intervention, 234–241. Cham, Switzerland: Springer.
Russakovsky, O., J. Deng, H. Su, J. Krause, S. Satheesh, S. Ma, Z. Huang, A. Karpathy, A. Khosla, and M. Bernstein. 2015. “ImageNet large scale visual recognition challenge.” Int. J. Comput. Vis. 115 (3): 211–252. https://doi.org/10.1007/s11263-015-0816-y.
Shorten, C., and T. M. Khoshgoftaar. 2019. “A survey on image data augmentation for deep learning.” J. Big Data 6 (1): 1–48. https://doi.org/10.1186/s40537-019-0197-0.
Simonyan, K., A. Vedaldi, and A. Zisserman. 2013. “Deep inside convolutional networks: Visualising image classification models and saliency maps.” Preprint, submitted December 20, 2013. http://arxiv.org/abs/1312.6034.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. http://arxiv.org/abs/1409.1556.
Soltani, M. M., Z. Zhu, and A. Hammad. 2016. “Automated annotation for visual recognition of construction resources using synthetic images.” Autom. Constr. 62 (Feb): 14–23. https://doi.org/10.1016/j.autcon.2015.10.002.
Summers, C., and M. J. Dinneen. 2019. “Improved mixed-example data augmentation.” In Proc., 2019 IEEE Winter Conf. on Applications of Computer Vision (WACV), 1262–1270. New York: IEEE.
Szegedy, C., W. Liu, Y. Jia, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. “Going deeper with convolutions.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1–9. New York: IEEE.
Wang, Z., Y. Zhang, K. M. Mosalam, Y. Gao, and S. L. Huang. 2022. “Deep semantic segmentation for visual understanding on construction sites.” Comput.-Aided Civ. Infrastruct. Eng. 37 (2): 145–162. https://doi.org/10.1111/mice.12701.
Wu, R., S. Yan, Y. Shan, Q. Dang, and G. Sun. 2015. “Deep image: Scaling up image recognition.” Preprint, submitted January 13, 2015. http://arxiv.org/abs/1501.02876.
Yang, J., P. Vela, J. Teizer, and Z. Shi. 2014. “Vision-based tower crane tracking for understanding construction activity.” J. Comput. Civ. Eng. 28 (1): 103–112. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000242.
Yasmina, D., R. Karima, and A. Ouahiba. 2018. “Traffic signs recognition with deep learning.” In Proc., 2018 Int. Conf. on Applied Smart Systems (ICASS), 1–5. New York: IEEE.
Yi, W., Y. Sun, and S. He. 2018. “Data augmentation using conditional GANs for facial emotion recognition.” In Proc., 2018 Progress in Electromagnetics Research Symp. (PIERS-Toyama), 710–714. New York: IEEE.
Yu, J., Z. Lin, J. Yang, X. Shen, X. Lu, and T. S. Huang. 2018. “Generative image inpainting with contextual attention.” In Proc., IEEE conf. on Computer Vision and Pattern Recognition, 5505–5514. New York: IEEE.
Zeng, X., C. Liu, Y.-S. Wang, W. Qiu, L. Xie, Y.-W. Tai, C.-K. Tang, and A. L. Yuille. 2019. “Adversarial attacks beyond the image space.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 4302–4311. New York: IEEE.
Zhang, C., C. C. Chang, and M. Jamshidi. 2020. “Concrete bridge surface damage detection using a single-stage detector.” Comput.-Aided Civ. Infrastruct. Eng. 35 (4): 389–409. https://doi.org/10.1111/mice.12500.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 3 • May 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Apr 11, 2021
Accepted: Nov 29, 2021
Published online: Feb 7, 2022
Published in print: May 1, 2022
Discussion open until: Jul 7, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Shengyuan Li, Yushan Le, Xuefeng Zhao, Style-Controlled Image Synthesis of Concrete Damages Based on Fusion of Convolutional Encoder and Attention-Enhanced Conditional Generative Adversarial Network, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-6007, 38, 6, (2024).
- Yeji Hong, Wei-Chih Chern, Tam V. Nguyen, Hubo Cai, Hongjo Kim, Semi-supervised domain adaptation for segmentation models on different monitoring settings, Automation in Construction, 10.1016/j.autcon.2023.104773, 149, (104773), (2023).
- Jinwoo Kim, Daeho Kim, SangHyun Lee, Seokho Chi, Hybrid DNN training using both synthetic and real construction images to overcome training data shortage, Automation in Construction, 10.1016/j.autcon.2023.104771, 149, (104771), (2023).
- Amirhossein Zaji, Zheng Liu, Gaozhi Xiao, Jatinder S. Sangha, Yuefeng Ruan, A survey on deep learning applications in wheat phenotyping, Applied Soft Computing, 10.1016/j.asoc.2022.109761, 131, (109761), (2022).