Unsupervised Domain Adaptation Approach for Vision-Based Semantic Understanding of Bridge Inspection Scenes without Manual Annotations
Publication: Journal of Bridge Engineering
Volume 29, Issue 2
Abstract
Deep learning-based (DL) visual recognition algorithms are widely investigated to enhance the accuracy, efficiency, and objectivity of the bridge inspection process, which is largely manual today. These algorithms typically require a large amount of training data, which consists of images and corresponding annotations. The manual preparation of such data sets is time-consuming, and more automated data generation approaches that are aided by synthetic environments suffer from domain gaps, which result in poor performance in real-world tasks. This study investigates an unsupervised domain adaptation (UDA) approach for visual recognition in bridge inspection scenes to reduce and eventually eliminate the need for time-consuming and inaccurate manual image annotations. A state-of-the-art UDA framework, termed DAFormer, is applied to the synthetic source domain data with full annotations and real-world target domain data with no or partial annotations. The synthetic data set in this study is designed to correlate with real-world data by incorporating the relevant design standards and practices into the modeling step. Compared with the source-only supervised learning approach (which performed poorly on real-world data), the UDA improved the performance to a level close to the supervised learning that used real-world data with manual annotations (the Intersection over Union (IoU) difference is only 1.03%). Furthermore, the UDA approach outperformed the supervised learning that used target domain data if the small amount of annotated target domain data is mixed with the synthetic source domain data to guide the network’s learning of patterns that only exist in the real-world environment (the IoU improvement was 5.03%). The UDA approach presented in this study facilitates the applications of DL-based visual recognition algorithms to bridge inspection tasks with limited manual effort.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
The authors would like to acknowledge the financial support of the National Natural Science Foundation of China (Grant No. 42250410334) and the Fundamental Research Funds for the Central Universities. The authors would like to thank Linlong Meng and Zeyi Liu, graduate students at Zhejiang University/University of Illinois Urbana-Champaign Institute, for their support in manually annotating real-world images used in this research.
References
Bao, Y., and H. Li. 2021. “Machine learning paradigm for structural health monitoring.” Struct. Health Monit. 20 (4): 1353–1372. https://doi.org/10.1177/1475921720972416/ASSET/IMAGES/10.1177_1475921720972416-IMG1.PNG.
Bentley. n.d. “Bridge design 1 software|Bentley systems.” Accessed August 9, 2023. https://www.bentley.com/software/bridge-design-1/.
Bianchi, E., and M. Hebdon. 2022. “Visual structural inspection datasets.” Autom. Constr. 139: 104299. https://doi.org/10.1016/J.AUTCON.2022.104299.
Chang, A. X., et al. 2015. “ShapeNet: An information-rich 3D model repository.” Preprint, submitted December 9, 2015. https://arxiv.org/abs/1512.03012v1.
Chen, L.-C., G. Papandreou, I. Kokkinos, K. Murphy, and A. L. Yuille. 2018. “Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected CRFs.” IEEE Trans. Pattern Anal. Mach. Intell. 40 (4): 834–848. https://doi.org/10.1109/TPAMI.2017.2699184.
Cheng, H., W. Chai, J. Hu, W. Ruan, M. Shi, H. Kim, Y. Cao, and Y. Narazaki. 2023. “Random bridge generator as a platform for developing computer vision-based structural inspection algorithms.” IET Cyber-syst. Robot.
Cordts, M., M. Omran, S. Ramos, T. Rehfeld, M. Enzweiler, R. Benenson, U. Franke, S. Roth, and B. Schiele. 2016. “The cityscapes dataset for semantic urban scene understanding.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 3213–3223. New York: Institute of Electrical and Electronics Engineers (IEEE).
Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. “Imagenet: A large-scale hierarchical image database.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 248–255. New York: Institute of Electrical and Electronics Engineers (IEEE).
Dong, C.-Z., and F. N. Catbas. 2021. “A review of computer vision–based structural health monitoring at local and global levels.” Struct. Health Monit. 20 (2): 692–743. https://doi.org/10.1177/1475921720935585.
Dosovitskiy, A., et al. 2020. “An image is worth 16(16 words: Transformers for image recognition at scale.” Preprint, submitted October 22, 2020. https://doi.org/10.48550/arxiv.2010.11929.
FHWA (Federal Highway Administration). 2004. National bridge inspection standards regulations. Washington, DC: FHWA.
Google Street View. n.d. “Bringing your map to life, one image at a time.” Accessed September 25, 2018. https://www.google.com/streetview/.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: Institute of Electrical and Electronics Engineers (IEEE).
Hoffman, J., E. Tzeng, T. Park, J.-Y. Zhu, P. Isola, K. Saenko, A. A. Efros, and T. Darrell. 2018. “CyCADA: Cycle-consistent adversarial domain adaptation.” In Proc., 35th Int. Conf. on Machine Learning, 1989–1998. Cambridge, MA: JMLR.
Hoffman, J., D. Wang, F. Yu, and T. Darrell. 2016. “FCNs in the wild: Pixel-level adversarial and constraint-based adaptation.” Preprint, submitted December 8, 2016. https://arxiv.org/abs/1612.02649v1.
Hoskere, V., Y. Narazaki, T. A. Hoang, and B. F. Spencer. 2020. “Madnet: Multi-task semantic segmentation of multiple types of structural materials and damage in images of civil infrastructure.” J. Civ. Struct. Health Monit. 10 (5): 757–773. https://doi.org/10.1007/s13349-020-00409-0.
Hoskere, V., Y. Narazaki, and B. F. Spencer. 2022. “Physics-based graphics models in 3D synthetic environments as autonomous vision-based inspection testbeds.” Sensors 22 (2): 532. https://doi.org/10.3390/S22020532.
Hoyer, L., D. Dai, and L. Van Gool. 2022. “DAFormer: Improving network architectures and training strategies for domain-adaptive semantic segmentation.” Preprint, submitted November 29, 2021. https://doi.org/10.48550/arXiv.2111.14887.
Huang, H., L. Lin, R. Tong, H. Hu, Q. Zhang, Y. Iwamoto, X. Han, Y.-W. Chen, and J. Wu. 2020. “UNet 3+: A full-scale connected UNet for medical image segmentation.” Preprint, submitted April 19, 2020. https://doi.org/10.48550/arXiv.2004.08790.
Isola, P., J.-Y. Zhu, T. Zhou, and A. A. Efros. 2017. “Image-To-Image translation with conditional adversarial networks.” Preprint, submitted November 21, 2016. https://doi.org/10.48550/arXiv.1611.07004.
Kim, H., Y. Narazaki, and B. F. Spencer Jr. 2023. “Automated bridge component recognition using close-range images from unmanned aerial vehicles.” Eng. Struct. 274: 115184. https://doi.org/10.1016/J.ENGSTRUCT.2022.115184.
Kim, H., J. Yoon, and S.-H. Sim. 2020. “Automated bridge component recognition from point clouds using deep learning.” Struct. Control Health Monit. 27 (9). e2591. https://doi.org/10.1002/stc.2591.
Kim, M., and H. Byun. 2020. “Learning texture invariant representation for domain adaptation of semantic segmentation.” Preprint, submitted March 2, 2020. https://doi.org/10.48550/arXiv.2003.00867.
Kono, M., and Y. Matsumoto. 1965. “Design of the standard rigid frame railway bridge in new Tokaido line.” Trans. Jpn Soc. Civ. Eng. 1965 (115): 13–25. https://doi.org/10.2208/jscej1949.1965.115_13.
Kuo, W., A. Angelova, T. Y. Lin, and A. Dai. 2020. “Mask2CAD: 3D shape prediction by learning to segment and retrieve.” In Proc., 16th European Conf. on Computer Vision, edited by A. Vedaldi, H. Bischof, T. Brox, and J.-M. Frahm, 260–277. Cham, Switzerland: Springer. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 12348 LNCS.
Labelbox. n.d. “Labelbox: The leading training data platform.” Accessed May 19, 2020. https://labelbox.com/.
Levine, N. M., Y. Narazaki, and B. F. Spencer. 2022. “Performance-based post-earthquake building evaluations using computer vision-derived damage observations.” Adv. Struct. Eng. 25 (16): 3425–3449. https://doi.org/10.1177/13694332221119883.
Levine, N. M., Y. Narazaki, and B. F. Spencer. 2023. “Development of a building information model-guided post-earthquake building inspection framework using 3D synthetic environments.” Earthquake Eng. Eng. Vibr. 22: 279–307. https://doi.org/10.1007/S11803-023-2167-Y.
Levine, N. M., and B. F. Spencer. 2022. “Post-earthquake building evaluation using UAVs: A BIM-based digital twin framework.” Sensors 22 (3): 873. https://doi.org/10.3390/S22030873.
Li, Y., L. Y. Microsoft, and N. Vasconcelos. 2019. “Bidirectional learning for domain adaptation of semantic segmentation.” Preprint, submitted April 24, 2019. https://doi.org/10.48550/arXiv.1904.10620.
Lichtenstein, A. G. 1993. “The silver bridge collapse recounted.” J. Perform. Constr. Facil 7 (4): 249–261. https://doi.org/10.1061/(ASCE)0887-3828(1993)7:4(249).
Lin, J. J., A. Ibrahim, S. Sarwade, and M. Golparvar-Fard. 2021. “Bridge inspection with aerial robots: Automating the entire pipeline of visual data capture, 3D mapping, defect detection, analysis, and reporting.” J. Comput. Civil Eng. 35 (2): 04020064. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000954.
Liu, Z., Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. 2021. “Swin transformer: Hierarchical vision transformer using shifted windows.” Preprint, submitted March 25, 2021. https://doi.org/10.48550/arXiv.2103.14030.
Long, J., E. Shelhamer, and T. Darrell. 2017. “Fully convolutional networks for semantic segmentation.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (4): 640–651. https://doi.org/10.1109/TPAMI.2016.2572683.
Loshchilov, I., and F. Hutter. 2017. “Decoupled weight decay regularization.” In Proc., 7th Int. Conf. on Learning Representations. Amherst, MA: OpenReview.net, University of Massachusetts Amherst.
midas Bridge. n.d. “midas Bridge: Bridge analysis and design software.” Accessed August 9, 2023. https://www.midasbridge.com/en/main.
MLIT (Ministry of Land, Infrastructure, Transport and Tourism). n.d.-a. “PLATEAU.” Accessed August 9, 2023. https://www.mlit.go.jp/plateau/.
MLIT (Ministry of Land, Infrastructure, Transport and Tourism). n.d.-b. “道路:道路の老朽化対策.” Accessed January 25, 2023. https://www.mlit.go.jp/road/sisaku/yobohozen/yobohozen.html.
MOT (Ministry of Transport of the People’s Republic of China). 2007. “关于印发《公路桥梁养护管理工作制度》的通知-政府信息公开-交通运输部.” https://xxgk.mot.gov.cn/2020/jigou/glj/202006/t20200623_3312762.html.
Narazaki, Y., V. Hoskere, G. Chowdhary, and B. F. Spencer. 2022. “Vision-based navigation planning for autonomous post-earthquake inspection of reinforced concrete railway viaducts using unmanned aerial vehicles.” Autom. Constr. 137: 104214. https://doi.org/10.1016/J.AUTCON.2022.104214.
Narazaki, Y., V. Hoskere, T. A. Hoang, Y. Fujino, A. Sakurai, and B. F. Spencer. 2020. “Vision-based automated bridge component recognition with high-level scene consistency.” Comput.-Aided Civ. Infrastruct. Eng. 35 (5): 465–482. https://doi.org/10.1111/mice.12505.
Narazaki, Y., V. Hoskere, K. Yoshida, B. F. Spencer, and Y. Fujino. 2021. “Synthetic environments for vision-based structural condition assessment of Japanese high-speed railway viaducts.” Mech. Syst. Sig. Process. 160: 107850. https://doi.org/10.1016/j.ymssp.2021.107850.
Olsson, V., W. Tranheden, J. Pinto, and L. Svensson. 2021. “ClassMix: Segmentation-based data augmentation for semi-supervised learning.” Preprint, submitted July 15, 2020. https://doi.org/10.48550/arXiv.2007.07936.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 779–788. New York: Institute of Electrical and Electronics Engineers (IEEE).
Ren, S., K. He, R. Gershick, and J. Sun. 2017. “Faster R-CNN: Towards real-time object detection with region proposal networks.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Richter, S. R., V. Vineet, S. Roth, and V. Koltun. 2016. “Playing for data: Ground truth from computer games.” In Proc., European Conf. on Computer Vision, 102–118. Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 9906 LNCS. Berlin, Germany: Springer Science.
Ros, G., L. Sellart, J. Materzynska, D. Vazquez, and A. M. Lopez. 2016. “The SYNTHIA dataset: A large collection of synthetic images for semantic segmentation of urban scenes.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 3234–3243. New York: Institute of Electrical and Electronics Engineers (IEEE).
Sakaridis, C., D. Dai, L. Van Gool, and E. Zürich. 2021. “ACDC: The adverse conditions dataset with correspondences for semantic driving scene understanding.” Accessed November 26, 2023. https://acdc.vision.ee.ethz.ch.
Spencer, B. F., V. Hoskere, and Y. Narazaki. 2019. “Advances in computer vision-based civil infrastructure inspection and monitoring.” Engineering 5 (2): 199–222. https://doi.org/10.1016/J.ENG.2018.11.030.
Toldo, M., A. Maracani, U. Michieli, and P. Zanuttigh. 2020. “Unsupervised domain adaptation in semantic segmentation: A review.” Technologies 8 (2): 35. https://doi.org/10.3390/TECHNOLOGIES8020035.
Tranheden, W., V. Olsson, J. Pinto, and L. Svensson. 2021. “DACS: Domain adaptation via cross-domain mixed sampling.” Preprint, submitted July 17, 2020. https://doi.org/10.48550/arXiv.2007.08702.
Wang, C.-Y., A. Bochkovskiy, and H.-Y. M. Liao. 2022a. “YOLOv7: Trainable bag-of-freebies sets new state-of-the-art for real-time object detectors.” Preprint, submitted July 6, 2022. https://doi.org/10.48550/arxiv.2207.02696.
Wang, J., C. Lan, C. Liu, Y. Ouyang, T. Qin, W. Lu, Y. Chen, W. Zeng, and P. Yu. 2022b. “Generalizing to unseen domains: A survey on domain generalization.” IEEE Trans. Knowl. Data Eng. 35: 8052–8072. https://doi.org/10.1109/TKDE.2022.3178128.
Wells, J., B. Lovelace, and I. Collins Engineers. 2018. Improving the quality of bridge inspections using Unmanned Aircraft Systems (UAS). Washington, DC: DOT, Bureau of Transportation Statistics.
Wells, J., B. Lovelace, and I. Collins Engineers. 2021. Unmanned aircraft systems (UAS) – metro district bridge inspection implementation. Rep. No. MN 2021-13. St. Paul, MN: MNDOT.
Xie, E., W. Wang, Z. Yu, A. Anandkumar, J. M. Alvarez, and P. Luo. 2021. “Segformer: Simple and efficient design for semantic segmentation with transformers.” Adv. Neural Inf. Process. Syst. 34: 12077–12090.
Xu, F. Y., M. J. Zhang, L. Wang, and J. R. Zhang. 2016. “Recent highway bridge collapses in China: Review and discussion.” J. Perform. Constr. Facil 30 (5): 04016030. https://doi.org/10.1061/(ASCE)CF.1943-5509.0000884.
Zhai, G., Y. Narazaki, S. Wang, S. A. V. Shajihan, and B. F. Spencer. 2022. “Synthetic data augmentation for pixel-wise steel fatigue crack identification using fully convolutional networks.” Smart Struct. Syst. 29 (1): 237–250. https://doi.org/10.12989/SSS.2022.29.1.237.
Zhu, J.-Y., T. Park, P. Isola, A. A. Efros, and B. A. Research. 2017. “Unpaired image-To-image translation using cycle-consistent adversarial networks.” In Proc., IEEE Int. Conf. on Computer Vision, 2223–2232. New York: Institute of Electrical and Electronics Engineers (IEEE).
Zou, Y., Y. Zhiding, B. V. K. Vijaya Kumar, and J. Wang. 2018. “Unsupervised domain adaptation for semantic segmentation via class-balanced self-training.” In Proc., 15th European Conf. on Computer Vision, edited by V. Ferrari, M. Hebert, C. Sminchisescu, and Y. Weiss, 289–305. Cham, Switzerland: Springer.
Information & Authors
Information
Published In
Journal of Bridge Engineering
Volume 29 • Issue 2 • February 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Jun 2, 2023
Accepted: Oct 10, 2023
Published online: Dec 5, 2023
Published in print: Feb 1, 2024
Discussion open until: May 5, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.