Comparative Effectiveness of Data Augmentation Using Traditional Approaches versus StyleGANs in Automated Sewer Defect Detection
Publication: Journal of Water Resources Planning and Management
Volume 149, Issue 9
Abstract
This study compared how traditional data augmentation and the state-of-the-art style-based generative adversarial network (StyleGAN) benefit automated sewer defect detection using a You Only Look Once (YOLO) object detection model. There were 25 augmentation scenarios (13 individual augmentations, 8 combined augmentations, and 3 StyleGAN augmentations, as well as a baseline) investigated, with 3 types of data set sizes (300, 600, and 1,200 images) and 4 types of sewer defects (disjoint, obstacle, residential wall, and tree root). Results showed that geometric transformations generally outperformed color transformations. Among the traditional methods, random cropping worked best to improve the detector performance, meaning that combined effects may not be as strong as a single type of augmentation. It was also noted that not all augmentation measures benefited the detection, as in our case, 24% of the augmentation effects resulted in lower detection accuracy than the baseline scenario. StyleGAN performed remarkably well in improving the data quality through increasing style and diversity. Moreover, data augmentation had a greater impact on improving the detection of residential wall and tree root [with an increase in the mean of average precision () of 32% and 22%] in comparison to disjoint and obstacle (21% and 16%), respectively. The findings will benefit the future selection and use of augmentation methods in enhancing the performance of deep learning models.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The training and testing data sets, models, and code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This research was funded by the Youth Promotion Program of the Natural Science Foundation of Guangdong Province, China (Grant No. 2023A1515030126), and the National Natural Science Foundation of China (Grant No. 51809049).
References
Butler, D., and J. Davies. 2010. Urban drainage. 3rd ed. London: CRC Press.
Cheng, J. C. P., and M. Wang. 2018. “Automated detection of sewer pipe defects in closed-circuit television images using deep learning techniques.” Autom. Constr. 95 (Nov): 155–171. https://doi.org/10.1016/j.autcon.2018.08.006.
Donahue, J., Y. Jia, O. Vinyals, J. Hoffman, N. Zhang, E. Tzeng, and T. Darrell. 2014. “DeCAF: A deep convolutional activation feature for generic visual recognition.” In Proc., Int. Conf. on Machine Learning, 647–655. Norfolk, MA: Journal of Machine Learning Research.
Duran, O., K. Althoefer, and L. D. Seneviratne. 2002. “Automated sewer pipe inspection through image processing.” In Proc., 2002 IEEE Int. Conf. on Robotics and Automation (Cat. No. 02CH37292), 2551–2556. New York: IEEE.
Fetty, L., M. Bylund, P. Kuess, G. Heilemann, T. Nyholm, D. Georg, and T. Löfstedt. 2020. “Latent space manipulation for high-resolution medical image synthesis via the StyleGAN.” Z. Med. Phys. 30 (4): 305–314. https://doi.org/10.1016/j.zemedi.2020.05.001.
Fokina, D., E. Muravleva, G. Ovchinnikov, and I. Oseledets. 2020. “Microstructure synthesis using style-based generative adversarial networks.” Phys. Rev. E 101 (4): 043308. https://doi.org/10.1103/PhysRevE.101.043308.
Frégier, Y., and J.-B. Gouray. 2021. “Mind2Mind: Transfer learning for GANs.” In Proc., Int. Conf. on Geometric Science of Information: 5th Int. Conf., GSI 2021, 851–859. Cham, Switzerland: Springer.
Fu, G., Y. Jin, S. Sun, Z. Yuan, and D. Butler. 2022. “The role of deep learning in urban water management: A critical review.” Water Res. 223 (Sep): 118973. https://doi.org/10.1016/j.watres.2022.118973.
Gao, Y., B. Kong, and K. M. Mosalam. 2019. “Deep leaf-bootstrapping generative adversarial network for structural image data augmentation.” Comput.-Aided Civ. Infrastruct. Eng. 34 (9): 755–773. https://doi.org/10.1111/mice.12458.
Gao, Y., and K. M. Mosalam. 2018. “Deep transfer learning for image-based structural damage recognition.” Comput.-Aided Civ. Infrastruct. Eng. 33 (9): 748–768. https://doi.org/10.1111/mice.12363.
Gong, A., X. Yao, and W. Lin. 2020. “Dermoscopy image classification based on StyleGANs and decision fusion.” IEEE Access 8 (Apr): 70640–70650. https://doi.org/10.1109/ACCESS.2020.2986916.
Guo, G., X. Yu, S. Liu, Z. Ma, Y. Wu, X. Xu, X. Wang, K. Smith, and X. Wu. 2021. “Leakage detection in water distribution systems based on time–frequency convolutional neural network.” J. Water Resour. Plann. Manage. 147 (2): 04020101. https://doi.org/10.1061/(ASCE)WR.1943-5452.0001317.
Guo, W., X. Zhang, D. Zhang, Z. Chen, B. Zhou, D. Huang, and Q. Li. 2022. “Detection and classification of pipe defects based on pipe-extended feature pyramid network.” Autom. Constr. 141 (Sep): 104399. https://doi.org/10.1016/j.autcon.2022.104399.
Halfawy, M. R., and J. Hengmeechai. 2014. “Automated defect detection in sewer closed circuit television images using histograms of oriented gradients and support vector machine.” Autom. Constr. 38 (Mar): 1–13. https://doi.org/10.1016/j.autcon.2013.10.012.
Halfawy, M. R., and J. Hengmeechai. 2015. “Integrated vision-based system for automated defect detection in sewer closed circuit television inspection videos.” J. Comput. Civ. Eng. 29 (1): 04014024. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000312.
Hassan, S. I., L. M. Dang, I. Mehmood, S. Im, C. Choi, J. Kang, Y.-S. Park, and H. Moon. 2019. “Underground sewer pipe condition assessment based on convolutional neural networks.” Autom. Constr. 106 (Oct): 102849. https://doi.org/10.1016/j.autcon.2019.102849.
Haurum, J. B., and T. B. Moeslund. 2021. “Sewer-ML: A multi-label sewer defect classification dataset and benchmark.” In Proc., 2021 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 13451–13462. New York: IEEE.
He, L., Z. Qiang, X. Shao, H. Lin, M. Wang, and F. Dai. 2022. “Research on high-resolution face image inpainting method based on StyleGAN.” Electronics 11 (10): 1620. https://doi.org/10.3390/electronics11101620.
Heusel, M., H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. 2017. “Gans trained by a two time-scale update rule converge to a local Nash equilibrium.” In Proc., Advances in Neural Information Processing Systems, 6626–6637. Red Hook, NY: Curran Associates.
Jurio, A., M. Pagola, M. Galar, C. Lopez-Molina, and D. Paternain. 2010. “A comparison study of different color spaces in clustering based image segmentation.” In Proc., Information Processing and Management of Uncertainty in Knowledge-Based Systems. Applications: 13th Int. Conf., IPMU 2010, 532–541. Berlin: Springer.
Karras, T., T. Aila, S. Laine, and J. Lehtinen. 2018. “Progressive growing of GANs for improved quality, stability, and variation.” In Proc., Int. Conf. on Learning Representations, 1–26. Amherst, MA: OpenReview.
Karras, T., S. Laine, and T. Aila. 2019. “A style-based generator architecture for generative adversarial networks.” In Proc., 2019 IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR), 4396–4405. New York: IEEE.
Kingma, D. P., and J. L. Ba. 2015. “Adam: A method for stochastic optimization.” In Proc., Int. Conf. on Learning Representations, 1–15. Amherst, MA: OpenReview.
Kokomoto, K., R. Okawa, K. Nakano, and K. Nozaki. 2021. “Intraoral image generation by progressive growing of generative adversarial network and evaluation of generated image quality by dentists.” Sci. Rep. 11 (1): 18517. https://doi.org/10.1038/s41598-021-98043-3.
Kramberger, T., and B. Potocnik. 2020. “LSUN-Stanford car dataset: Enhancing large-scale car image datasets using deep learning for usage in GAN training.” Appl. Sci. 10 (14): 4913. https://doi.org/10.3390/app10144913.
Kumar, S. S., and D. M. Abraham. 2019. “A deep learning based automated structural defect detection system for sewer pipelines.” In Proc., Computing in Civil Engineering 2019: Smart Cities, Sustainability, and Resilience, 226–233. Reston, VA: ASCE.
LeCun, Y., Y. Bengio, and G. Hinton. 2015. “Deep learning.” Nature 521 (7553): 436–444. https://doi.org/10.1038/nature14539.
Lemley, J., S. Bazrafkan, and P. Corcoran. 2017. “Smart augmentation learning an optimal data augmentation strategy.” IEEE Access 5 (Apr): 5858–5869. https://doi.org/10.1109/ACCESS.2017.2696121.
Li, D., A. Cong, and S. Guo. 2019. “Sewer damage detection from imbalanced CCTV inspection data using deep convolutional neural networks with hierarchical classification.” Autom. Constr. 101 (May): 199–208. https://doi.org/10.1016/j.autcon.2019.01.017.
Li, D., Q. Xie, Z. Yu, Q. Wu, J. Zhou, and J. Wang. 2021. “Sewer pipe defect detection via deep learning with local and global feature fusion.” Autom. Constr. 129 (Sep): 103823. https://doi.org/10.1016/j.autcon.2021.103823.
Lin, M. 2014. “Health inspection and analysis of sewer system in an area of Fuzhou City.” China Water & Waste Water 30 (9): 96–98. https://doi.org/10.19853/j.zgjsps.1000-4602.2014.09.024.
Lin, Y.-Z., Z.-H. Nie, and H.-W. Ma. 2017. “Structural damage detection with automatic feature-extraction through deep learning.” Comput.-Aided Civ. Infrastruct. Eng. 32 (12): 1025–1046. https://doi.org/10.1111/mice.12313.
Ma, D., H. Fang, N. Wang, H. Lu, J. Matthews, and C. Zhang. 2023. “Transformer-optimized generation, detection, and tracking network for images with drainage pipeline defects.” Comput.-Aided Civ. Infrastruct. Eng. 1 (1): 1–19. https://doi.org/10.1111/mice.12970.
Ma, D., J. Liu, H. Fang, N. Wang, C. Zhang, Z. Li, and J. Dong. 2021. “A multi-defect detection system for sewer pipelines based on StyleGAN-SDM and fusion CNN.” Constr. Build. Mater. 312 (Dec): 125385. https://doi.org/10.1016/j.conbuildmat.2021.125385.
Meijer, D., L. Scholten, F. Clemens, and A. Knobbe. 2019. “A defect classification methodology for sewer image sets with convolutional neural networks.” Autom. Constr. 104 (Aug): 281–298. https://doi.org/10.1016/j.autcon.2019.04.013.
Oh, C., L. M. Dang, D. Han, and H. Moon. 2022. “Robust sewer defect detection with text analysis based on deep learning.” IEEE Access 10 (Apr): 46224–46237. https://doi.org/10.1109/ACCESS.2022.3168660.
Oquab, M., L. Bottou, I. Laptev, and J. Sivic. 2014. “Learning and transferring mid-level image representations using convolutional neural networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1717–1724. New York: IEEE.
Pan, G., Y. Zheng, S. Guo, and Y. Lv. 2020. “Automatic sewer pipe defect semantic segmentation based on improved U-Net.” Autom. Constr. 119 (Nov): 103383. https://doi.org/10.1016/j.autcon.2020.103383.
Perez, F., C. Vasconcelos, S. Avila, and E. Valle. 2018. “Data augmentation for skin lesion analysis.” In Proc., OR 2.0 Context-Aware Operating Theaters, Computer Assisted Robotic Endoscopy, Clinical Image-Based Procedures, and Skin Image Analysis, 303–311. Cham, Switzerland: Springer.
Radford, A., L. Metz, and S. Chintala. 2016. “Unsupervised representation learning with deep convolutional generative adversarial networks.” In Proc., Int. Conf. on Learning Representations 2016, 1–16. New York: IEEE.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., 2016 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 779–788. New York: IEEE.
Redmon, J., and A. Farhadi. 2017. “YOLO9000: Better, faster, stronger.” In Proc., 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 6517–6525. New York: IEEE.
Ren, S., K. He, R. Girshick, and J. Sun. 2017. “Faster R-CNN: Towards real-time object detection with region proposal networks.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Shin, H.-C., H. R. Roth, M. Gao, L. Lu, Z. Xu, I. Nogues, J. Yao, D. Mollura, and R. M. Summers. 2016. “Deep convolutional neural networks for computer-aided detection: CNN architectures, dataset characteristics and transfer learning.” IEEE Trans. Med. Imaging 35 (5): 1285–1298. https://doi.org/10.1109/TMI.2016.2528162.
Shorten, C., and T. M. Khoshgoftaar. 2019. “A survey on image data augmentation for deep learning.” J. Big Data 6 (1): 60. https://doi.org/10.1186/s40537-019-0197-0.
Situ, Z., S. Teng, X. Liao, G. Chen, and Q. Zhou. 2023. “Real-time sewer defect detection based on YOLO network, transfer learning, and channel pruning algorithm.” J. Civ. Struct. Health Monit. 1 (1): 1–17. https://doi.org/10.1007/s13349-023-00681-w.
Song, S., K. Chang, K. Yun, C. Jun, and J.-G. Baek. 2022. “Defect synthesis using latent mapping adversarial network for automated visual inspection.” Electronics 11 (17): 2763. https://doi.org/10.3390/electronics11172763.
Strelcenia, E., and S. Prakoonwit. 2023. “A survey on GAN techniques for data augmentation to address the imbalanced data issues in credit card fraud detection.” Mach. Learn. Knowl. Extr. 5 (1): 304–329. https://doi.org/10.3390/make5010019.
Tan, Y., R. Cai, J. Li, P. Chen, and M. Wang. 2021. “Automatic detection of sewer defects based on improved you only look once algorithm.” Autom. Constr. 131 (Nov): 103912. https://doi.org/10.1016/j.autcon.2021.103912.
Tellez, D., G. Litjens, P. Bándi, W. Bulten, J.-M. Bokhorst, F. Ciompi, and J. Van Der Laak. 2019. “Quantifying the effects of data augmentation and stain color normalization in convolutional neural networks for computational pathology.” Med. Image Anal. 58 (Dec): 101544. https://doi.org/10.1016/j.media.2019.101544.
Tzeng, E., J. Hoffman, T. Darrell, and K. Saenko. 2015. “Simultaneous deep transfer across domains and tasks.” In Proc., Int. Conf. on Computer Vision, 4068–4076. Northwest Washington, DC: IEEE Computer Society.
Urbonas, A., V. Raudonis, R. Maskeliunas, and R. Damasevicius. 2019. “Automated identification of wood veneer surface defects using faster region-based convolutional neural network with data augmentation and transfer learning.” Appl. Sci. 9 (22): 4898. https://doi.org/10.3390/app9224898.
Varkarakis, V., S. Bazrafkan, and P. Corcoran. 2020. “Deep neural network and data augmentation methodology for off-axis iris segmentation in wearable headsets.” Neural Networks 121 (Jan): 101–121. https://doi.org/10.1016/j.neunet.2019.07.020.
Volpi, R., H. Namkoong, O. Sener, J. C. Duchi, V. Murino, and S. Savarese. 2018. “Generalizing to unseen domains via adversarial data augmentation.” In Vol. 31 of Proc., Advances in Neural Information Processing Systems, 1–11. Red Hook, NY: Curran Associates.
Wang, M., S. S. Kumar, and J. C. P. Cheng. 2021. “Automated sewer pipe defect tracking in CCTV videos based on defect detection and metric learning.” Autom. Constr. 121 (Jan): 103438. https://doi.org/10.1016/j.autcon.2020.103438.
Wang, Y., A. Gonzalez-Garcia, D. Berga, L. Herranz, F. S. Khan, and J. V. D. Weijer. 2020. “MineGAN: Effective knowledge transfer from GANs to target domains with few images.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 9332–9341. New York: IEEE.
Wang, Y., C. Wu, L. Herranz, J. van de Weijer, A. Gonzalez-Garcia, and B. Raducanu. 2018. “Transferring GANs: Generating images from limited data.” In Proc., European Conf. on Computer Vision (ECCV), 218–234. Berlin: Springer-Verlag.
Wirahadikusumah, R., D. Abraham, and T. Iseley. 2001. “Challenging issues in modeling deterioration of combined sewers.” J. Infrastruct. Syst. 7 (2): 77–84. https://doi.org/10.1061/(ASCE)1076-0342(2001)7:2(77).
Xie, Q., D. Li, J. Xu, Z. Yu, and J. Wang. 2019. “Automatic detection and classification of sewer defects via hierarchical deep learning.” IEEE Trans. Autom. Sci. Eng. 16 (4): 1836–1847. https://doi.org/10.1109/TASE.2019.2900170.
Yang, M. D., and T. C. Su. 2008. “Automated diagnosis of sewer pipe defects based on machine learning approaches.” Expert Syst. Appl. 35 (3): 1327–1337. https://doi.org/10.1016/j.eswa.2007.08.013.
Ye, X., J. E. Zuo, R. Li, Y. Wang, L. Gan, Z. Yu, and X. Hu. 2019. “Diagnosis of sewer pipe defects on image recognition of multi-features and support vector machine in a southern Chinese city.” Front. Environ. Sci. Eng. 13 (Apr): 1–13. https://doi.org/10.1007/s11783-019-1102-y.
Yin, X., Y. Chen, A. Bouferguene, H. Zaman, M. Al-Hussein, and L. Kurach. 2020. “A deep learning-based framework for an automated defect detection system for sewer pipes.” Autom. Constr. 109 (Jan): 102967. https://doi.org/10.1016/j.autcon.2019.102967.
Zeiler, M. D., and R. Fergus. 2014. “Visualizing and understanding convolutional networks.” In Proc., European Conf. on Computer Vision, 818–833. Cham, Switzerland: Springer.
Zhang, C. B., C. C. Chang, and M. Jamshidi. 2020. “Concrete bridge surface damage detection using a single-stage detector.” Comput.-Aided Civ. Infrastruct. Eng. 35 (4): 389–409. https://doi.org/10.1111/mice.12500.
Zhang, Q., Z. Y. Wu, M. Zhao, J. Qi, Y. Huang, and H. Zhao. 2016. “Leakage zone identification in large-scale water distribution systems using multiclass support vector machines.” J. Water Resour. Plann. Manage. 142 (11): 04016042. https://doi.org/10.1061/(ASCE)WR.1943-5452.0000661.
Zhao, J., X. Hou, M. Pan, and H. Zhang. 2022. “Attention-based generative adversarial network in medical imaging: A narrative review.” Comput. Biol. Med. 149 (Oct): 105948. https://doi.org/10.1016/j.compbiomed.2022.105948.
Zhong, K., S. Teng, G. Liu, G. Chen, and F. Cui. 2020. “Structural damage features extracted by convolutional neural networks from mode shapes.” Appl. Sci. 10 (12): 4247. https://doi.org/10.3390/app10124247.
Zhou, Q., Z. Situ, S. Teng, W. Chen, G. Chen, and J. Su. 2022. “Comparison of classic object-detection techniques for automated sewer defect detection.” J. Hydroinf. 24 (2): 406–419. https://doi.org/10.2166/hydro.2022.132.
Zhu, J., and J. Song. 2020. “An intelligent classification model for surface defects on cement concrete bridges.” Appl. Sci. 10 (3): 972. https://doi.org/10.3390/app10030972.
Information & Authors
Information
Published In
Journal of Water Resources Planning and Management
Volume 149 • Issue 9 • September 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Apr 20, 2022
Accepted: May 24, 2023
Published online: Jul 10, 2023
Published in print: Sep 1, 2023
Discussion open until: Dec 10, 2023
ASCE Technical Topics:
- Automation and robotics
- Comparative studies
- Defects and imperfections
- Design (by type)
- Ecosystems
- Engineering fundamentals
- Environmental engineering
- Geometrics
- Highway and road design
- Infrastructure
- Lifeline systems
- Materials characterization
- Materials engineering
- Methodology (by type)
- Research methods (by type)
- Sewers
- Structural engineering
- Structural members
- Structural systems
- Systems engineering
- Trees
- Vegetation
- Walls
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.