Automated Bridge Inspection Image Retrieval Based on Deep Similarity Learning and GPS
Publication: Journal of Structural Engineering
Volume 150, Issue 3
Abstract
The inspection of highway bridge structures in the US is a task critical to the national transportation system. Inspection images contain abundant visual information that can be exploited to streamline bridge assessment and management tasks. However, historical inspection images often go unused in subsequent assessments because they tend to be disorganized and unlabeled. Further, due to the lack of global positioning system (GPS) metadata and visual ambiguity, it is often difficult for other inspectors to identify the location on the bridge where past images were taken. Although many approaches are being considered toward fully automated or semiautomated methods for bridge inspection, there are research opportunities to develop practical tools for inspectors to make use of those images already in a database. In this study, a deep learning–based image similarity technique is developed and combined with image geolocation data to localize and retrieve historical inspection images based on a current query image. A Siamese convolutional neural network (SCNN) is trained and validated on a gathered data set of over 1,000 real world bridge deck images collected by the Indiana Department of Transportation. A composite similarity (CS) metric is created for effective image ranking, and the overall method is validated on a subset of eight bridge’s images. The results show promise for implementation into existing databases and for other similar structural inspections, showing up to an 11-fold improvement in successful image retrieval when compared with random image selection.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request. Some or all data, models, or code used during the study were provided by a third party. Direct requests for these materials may be made to the provider as indicated in the Acknowledgements.
Acknowledgments
The authors would like to thank the Indiana Department of Transportation for providing the inspection images used in this study, and for providing permission to share these images and findings. The authors declare no funding or potential conflicts of interest with this work.
References
AASHTO. 2013. AASHTO manual for bridge element inspection. Washington, DC: AASHTO.
Abdallah, A. M., R. A. Atadero, and M. E. Ozbek. 2022. “A state-of-the-art review of bridge inspection planning: Current situation and future needs.” J. Bridge Eng. 27 (2): 03121001. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001812.
Alayoub, A., S. A. E. Rahim, S. Mustapha, D. Salam, A. Tehrani, and N. L. D. Khoa. 2022. “The application of machine learning to paint condition assessment using hyperspectral data.” In Proc., 12th Hyperspectral Imaging and Signal Processing: Evolution in Remote Sensing, 1–6. New York: IEEE. https://doi.org/10.1109/WHISPERS56178.2022.9955099.
Apple. 2021. “Recognizing people in photos through private on-device machine learning.” Accessed February 15, 2023. https://machinelearning.apple.com/research/recognizing-people-photos.
ASCE. 2021. “2021 report card for America’s infrastructure: Bridges.” Accessed February 3, 2023. https://infrastructurereportcard.org/cat-item/bridges-infrastructure/.
Bianchi, E. L., N. Sakib, C. Woolsey, and M. Hebdon. 2022. “Bridge inspection component registration for damage evolution.” Struct. Health Monit. 22 (Jun): 472–495. https://doi.org/10.1177/14759217221083647.
Cao, Q., Y. Ying, and P. Li. 2013. “Similarity metric learning for face recognition.” In Proc., IEEE Int. Conf. on Computer Vision, 2408–2415. New York: IEEE. https://doi.org/10.1109/ICCV.2013.299.
Cha, Y. J., W. Choi, and O. Büyüköztürk. 2017. “Deep learning-based crack damage detection using convolutional neural networks.” Comput. Aided Civ. Inf. 32 (5): 361–378. https://doi.org/10.1111/mice.12263.
Choi, J., J. A. Park, S. J. Dyke, C. M. Yeum, A. Lenjani, and I. Bilionis. 2022. “Similarity learning to enable building searches in post-event image data.” Comput.-Aided Civ. Infrastruct. Eng. 37 (2): 261–275. https://doi.org/10.1111/mice.12698.
Chun, P. J., T. Yamane, and Y. Maemura. 2022. “A deep learning-based image captioning method to automatically generate comprehensive explanations of bridge damage.” Comput.-Aided Civ. Infrastruct. Eng. 37 (11): 1387–1401. https://doi.org/10.1111/mice.12793.
Deng, J., A. Singh, Y. Zhou, Y. Lu, and C. S. Lee. 2022. “Review on computer vision-based crack detection and quantification methodologies for civil structures.” Constr. Build. Mater. 356 (Jun): 129238. https://doi.org/10.1016/j.conbuildmat.2022.129238.
Dorafshan, S., and M. Maguire. 2018. “Bridge inspection: Human performance, unmanned aerial systems and automation.” J. Civ. Struct. Health Monit. 8 (3): 443–476. https://doi.org/10.1007/s13349-018-0285-4.
El-Naqa, I., Y. Yang, N. P. Galatsanos, R. M. Nishikawa, and M. N. Wernick. 2004. “A similarity learning approach to content-based image retrieval: Application to digital mammography.” IEEE Trans. on Med. Imaging 23 (10): 1233–1244. https://doi.org/10.1109/TMI.2004.834601.
FHWA (Federal Highway Administration). 2022. Specifications for the national bridge inventory. Washington, DC: FHWA.
Gagliardi, V. F., G. Bella, R. Previti, and L. Menghini. 2022. “Automatic damage detection of bridge joints and road pavements by artificial neural networks ANNs.” In Proc., SPIE Remote Sensing, 12268. Berlin: SPIE. https://doi.org/10.1117/12.2636217.
Google. 2023. Google maps. Accessed January 12, 2023. https://www.google.com/maps.
Graybeal, B. A., B. M. Phares, D. D. Rolander, M. Moore, and G. Washer. 2002. “Visual inspection of highway bridges.” J. Nondestruct. Eval. 21 (3): 67–83. https://doi.org/10.1023/A:1022508121821.
Hadsell, R., S. Chopra, and Y. LeCun. 2006. “Dimensionality reduction by learning an invariant mapping.” In Proc., Computer Society Conf. on Computer Vision and Pattern Recognition, 1735–1742. New York: IEEE. https://doi.org/10.1109/CVPR.2006.100.
Howard, A., et al. 2019. “Searching for MobileNetV3.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 1314–1324. New York: IEEE. https://doi.org/10.1109/ICCV.2019.00140.
Jiang, S., J. Zhang, W. Wang, and Y. Wang. 2023. “Automatic inspection of bridge bolts using unmanned aerial vision and adaptive scale unification-based deep learning.” Remote Sens. 15 (2): 328. https://doi.org/10.3390/rs15020328.
Jiang, X., A. K. Gramopadhye, and B. J. Melloy. 2004. “Theoretical issues in the design of visual inspection systems.” Theor. Issues Ergon. Sci. 5 (3): 232–247. https://doi.org/10.1080/1463922021000050005.
Kao, S. P., Y. C. Chang, and F. L. Wang. 2023. “Combining the YOLOv4 deep learning model with UAV imagery processing technology in the extraction and quantization of cracks in bridges.” Sensors 23 (5): 2572. https://doi.org/10.3390/s23052572.
Kim, I. H., H. Jeon, S. C. Baek, W. H. Hong, and H. J. Jung. 2018. “Application of crack identification techniques for an aging concrete bridge inspection using an unmanned aerial vehicle.” Sensors 18 (6): 1881. https://doi.org/10.3390/s18061881.
Kong, X., K. Li, Y. Zhang, and S. Das. 2022. “Bridge deck deterioration: Reasons and patterns.” Transp. Res. Rec. 2676 (7): 570–584. https://doi.org/10.1177/03611981221080140.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2017. “Imagenet classification with deep convolutional neural networks.” Commun. ACM 60 (6): 84–90. https://doi.org/10.1145/3065386.
Ktena, S. I., S. Parisot, E. Ferrante, M. Rajchl, M. Lee, B. Glocker, and D. Rueckert. 2017. “Distance metric learning using graph convolutional networks: Application to functional brain networks.” In Vol. 10433 of Proc., Medical Image Computing and Computer Assisted Intervention–MCCAI 2017, 469–477. New York: Springer. https://doi.org/10.1007/978-3-319-66182-7_54.
Li, Z. J., K. Adamu, K. Yan, X. L. Xu, P. Shao, X. H. Li, and H. M. Bashir. 2022. “Detection of nut–bolt loss in steel bridges using deep learning techniques.” Sustainability 14 (17): 10837. https://doi.org/10.3390/su141710837.
Lim, R. S., H. M. La, and W. Sheng. 2014. “A robotic crack inspection and mapping system for bridge deck maintenance.” IEEE Trans. Autom. 11 (2): 367–378. https://doi.org/10.1109/TASE.2013.2294687.
Lin, J. J., A. Ibrahim, S. Sarwade, and M. Golparvar-Fard. 2021. “Bridge inspection with aerial robots: Automating the entire pipeline of visual data capture, 3D mapping, defect detection, analysis, and reporting.” J. Comput. Civ. Eng. 35 (2): 04020064. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000954.
Lovelace, B., and L. Wells. 2018. Improving the quality of bridge inspections using unmanned aircraft systems (UAS): Research Project Final Report. St. Paul, MN: Minnesota DOT.
Merry, K., and P. Bettinger. 2019. “Smartphone GPS accuracy study in an urban environment.” PLoS One 14 (7): e0219890. https://doi.org/10.1371/journal.pone.0219890.
Montes, K. M., J. Dang, J. Liu, and P. Chun. 2022. “Bridge status realization and management enhanced by UAV, SfM, and deep learning.” In Vol. 2 of Proc., European Workshop on Structural Health Monitoring, 536–545. New York: Springer. https://doi.org/10.1007/978-3-031-07258-1_55.
Nguyen, A., V. Gharehbaghi, N. T. Le, L. Sterling, U. I. Chaudhry, and S. Crawford. 2023. “ASR crack identification in bridges using deep learning and texture analysis.” In Vol. 50 Structures, 494–507. Amsterdam, Netherlands: Elsevier. https://doi.org/10.1016/j.istruc.2023.02.042.
Perry, B. J., Y. Guo, R. Atadero, and J. W. van de Lindt. 2020. “Streamlined bridge inspection system utilizing unmanned aerial vehicles (UAVs) and machine learning.” Measurement 164 (Jun): 108048. https://doi.org/10.1016/j.measurement.2020.108048.
Phares, B. M., G. A. Washer, D. D. Rolander, and B. A. Graybeal. 2004. “Routine highway bridge inspection condition documentation accuracy and reliability.” J. Bridge Eng. 9 (4): 403–413. https://doi.org/10.1061/(ASCE)1084-0702(2004)9:4(403).
Phillips, S., and S. Narasimhan. 2019. “Automating data collection for robotic bridge inspections.” J. Bridge Eng. 24 (8): 04019075. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001442.
Schroff, F., D. Kalenichenko, and J. Philbin. 2015. “FaceNet: A unified embedding for face recognition and clustering.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 815–823. New York: IEEE. https://doi.org/10.1109/CVPR.2015.7298682.
Seo, J., E. Jeong, and J. P. Wacker. 2022. “Machine learning approach to visual bridge inspection with drones.” In Proc., Structures Congress 2022. Reston, VA: ASCE. https://doi.org/10.1061/9780784484180.013.
Shariff, A., J. F. Bonnefon, and I. Rahwan. 2017. “Psychological roadblocks to the adoption of self-driving vehicles.” Nat. Hum. Behav. 1 (10): 694–696. https://doi.org/10.1038/s41562-017-0202-6.
Spencer, B. F., V. Hoskere, and Y. Narazaki. 2019. “Advances in computer vision-based civil infrastructure inspection and monitoring.” Engineering 5 (2): 199–222. https://doi.org/10.1016/j.eng.2018.11.030.
Sutter, B., A. Lelevé, M. T. Pham, O. Gouin, N. Jupille, M. Kuhn, P. Lulé, P. Michaud, and P. Rémy. 2018. “A semi-autonomous mobile robot for bridge inspection.” Autom. Constr. 91 (May): 111–119. https://doi.org/10.1016/j.autcon.2018.02.013.
Wing, M. G., A. Eklund, and L. D. Kellogg. 2005. “Consumer-grade global positioning system (GPS) accuracy and reliability.” J. For. 103 (4): 169–173. https://doi.org/10.1093/jof/103.4.169.
Yamane, T., P. J. Chun, J. Dang, and R. Honda. 2023. “Recording of bridge damage areas by 3D integration of multiple images and reduction of the variability in detected results.” Comput.-Aided Civ. Infrastruct. Eng. 38 (17): 2391–2407. https://doi.org/10.1111/mice.12971.
Yang, L., R. Jin, L. Mummert, R. Sukthankar, A. Goode, B. Zheng, S. O. H. Hoi, and M. Satyanarayanan. 2010. “A boosting framework for visuality-preserving distance metric learning and its application to medical image retrieval.” IEEE Trans. Pattern Anal. Mach. Intell. 32 (1): 30–44. https://doi.org/10.1109/TPAMI.2008.273.
Yang, T., Y. Liu, Y. Huang, J. Liu, and S. Wang. 2023. “Symmetry-driven unsupervised abnormal object detection for railway inspection.” IEEE Trans. Ind. Inform. 19 (12): 11487–11498. https://doi.org/10.1109/TII.2023.3246995.
Yeum, C. M., S. J. Dyke, B. Benes, J. Hacker, R. Ramirez, A. Lund, and S. Pujol. 2019. “Post event reconnaissance image documentation using automated classification. J. Perform. Constr. Facil. 33 (1): 04018103. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001253.
Yu, W., and M. Nishio. 2022. “Multilevel structural components detection and segmentation toward computer vision-based bridge inspection.” Sensors 22 (9): 3502. https://doi.org/10.3390/s22093502.
Zhang, L., F. Yang, Y. D. Zhang, and Y. J. Zhu. 2016. “Road crack detection using deep convolutional neural network.” In Proc., IEEE Int. Conf. on Image Processing (ICIP), 3708–3712. New York: IEEE. https://doi.org/10.1109/ICIP.2016.7533052.
Zhang, X., et al. 2022a. “Machine-supported bridge inspection image documentation using artificial intelligence.” Transp. Res. Rec. 2677 (5): 720–736. https://doi.org/10.1177/03611981221135803.
Zhang, X., C. Beck, A. Lenjani, L. Bonthron, A. Lund, X. Liu, S. J. Dyke, J. Ramirez, P. Baah, and J. Hunter. 2022b. “Enabling rapid large-scale seismic bridge vulnerability assessment through artificial intelligence.” Transp. Res. Rec. 2677 (2): 1354–1372. https://doi.org/10.1177/03611981221112950.
Information & Authors
Information
Published In
Journal of Structural Engineering
Volume 150 • Issue 3 • March 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Mar 30, 2023
Accepted: Oct 9, 2023
Published online: Dec 27, 2023
Published in print: Mar 1, 2024
Discussion open until: May 27, 2024
ASCE Technical Topics:
- Artificial intelligence and machine learning
- Bridge engineering
- Bridge management
- Bridges
- Bridges (by type)
- Computer programming
- Computing in civil engineering
- Construction engineering
- Construction management
- Geomatics
- Global navigation satellite systems
- Highway and road management
- Highway and road structures
- Highway bridges
- Highway transportation
- Infrastructure
- Inspection
- Intelligent transportation systems
- Neural networks
- Structural engineering
- Surveying methods
- Transportation engineering
- Transportation management
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.