Point Cloud–Based Concrete Surface Defect Semantic Segmentation
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 2
Abstract
Visual inspection is one of the main approaches for annual bridge inspection. Light detection and ranging (LiDAR) scanning is a new technology, which is beneficial because it collects the point clouds and the third dimension of the scanned objects. Deep learning (DL)-based methods have attracted researchers’ attention for concrete surface defect detection. However, no point cloud–based DL method currently is available for semantic segmentation of bridge surface defects without converting the data set into other representations, which results in increasing the size of the data set. Moreover, most of the current point cloud–based concrete surface defect detection methods focus on only one type of defect. On the other hand, a data set plays a key role in DL. Therefore, the lack of publicly available point cloud data sets for bridge surface defects is one of the reasons for the lack of studies in this area. To address these issues, this paper created a publicly available point cloud data set for concrete bridge surface defect detection, and developed a point cloud–based semantic segmentation DL method to detect different types of concrete surface defects. Surface Normal Enhanced PointNet++ (SNEPointNet++) was developed for semantic segmentation of concrete bridge surface defects (i.e., cracks and spalls). SNEPointNet++ focuses on two main characteristics related to surface defects (i.e., normal vector and depth) and considers the issues related to the data set (i.e., imbalanced data set). The data set, which was collected from four concrete bridges and classified into three classes (cracks, spalls, and no defect), is made available for other researchers. The model was trained and evaluated using 60% and 20% of the data set, respectively. Testing on the remaining part of the data set resulted in 93% and 92% recall for cracks and spalls, respectively. Spalls of the segments deeper than 7 cm (severe spalls) can be detected with 99% recall.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code generated or used during the study are available in a repository or online in accordance with funder data retention policies (https://github.com/neshatbln/SNEPointNet2).
Acknowledgments
The authors thank Ali Ghelmani for his help and contribution in implementation of SNEPointNet++.
References
Aryan, A., F. Bosché, and P. Tang. 2021. “Planning for terrestrial laser scanning in construction: A review.” Autom. Constr. 125 (May): 103551. https://doi.org/10.1016/j.autcon.2021.103551.
Bahreini, F., and A. Hammad. 2021. “Point cloud semantic segmentation of concrete surface defects using dynamic graph CNN.” In Vol. 38 of Proc., 38th Int. Symp. on Automation and Robotics in Construction, 379–386. Dubai, United Arab Emirates: IAARC Publication.
Bello, S. A., S. Yu, C. Wang, J. M. Adam, and J. Li. 2020. “Deep learning on 3D point clouds.” Remote Sens. 12 (11): 1729. https://doi.org/10.3390/rs12111729.
Bolourian, N. 2022. “Surface normal enhanced PointNet2.” Accessed May 12, 2022. https://github.com/neshatbln/SNEPointNet2.
Boulch, A., J. Guerry, B. Le Saux, and N. Audebert. 2018. “SnapNet: 3D point cloud semantic labeling with 2D deep segmentation networks.” Comput. Graph. 71 (Apr): 189–198. https://doi.org/10.1016/j.cag.2017.11.010.
Chen, X., and J. Li. 2016. “A feasibility study on use of generic mobile laser scanning system for detecting asphalt pavement cracks.” In Proc., Int. Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, ISPRS Arch, 545–549. Göttingen, Germany: Copernicus Publications. https://doi.org/10.5194/isprsarchives-XLI-B1-545-2016.
Chow, J. K., K. Liu, P. S. Tan, Z. S. Su, J. Wu, Z. Li, and Y.-H. Wang. 2021. “Automated defect inspection of concrete structure.” Autom. Constr. 132 (Dec): 103959. https://doi.org/10.1016/j.autcon.2021.103959.
Compute Canada. 2021. “Compute Canada.” Accessed January 10, 2021. https://ccdb.computecanada.ca.
Cui, Y., M. Jia, T.-Y. Lin, Y. Song, and S. Belongie. 2019. “Class-balanced loss based on effective number of samples.” In Proc., IEEECVF Conf. on Computer Vision and Pattern Recognition. Recognition, 9268–9277. New York: IEEE.
Dawood, T., Z. Zhu, and T. Zayed. 2017. “Machine vision-based model for spalling detection and quantification in subway networks.” Autom. Constr. 81 (Sep): 149–160. https://doi.org/10.1016/j.autcon.2017.06.008.
Demir, N., and E. Baltsavias. 2012. “Automated modeling of 3D building roofs using image and LiDAR data.” In Proc., Int. Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 35–40. Hannover, Germany: International Society for Photogrammetry and Remote Sensing.
Dorafshan, S., and M. Maguire. 2018. “Bridge inspection: Human performance, unmanned aerial systems and automation.” J. Civ. Struct. Health Monit. 8 (3): 443–476. https://doi.org/10.1007/s13349-018-0285-4.
FARO Technologies. 2011. User manual FARO laser scanner focus 3D. Lake Mary, FL: FARO Technologies.
FARO Technologies. 2017. “Quality setting function on the Focus3D.” Accessed May 24, 2021. https://knowledge.faro.com/Hardware/3D_Scanners/Focus/Quality_Setting_Function_on_the_Focus3D.
Gagnon, M., V. Gaudreault, and D. Overton. 2008. Age of public infrastructure: A provincial perspective. Ottawa: Statistics Canada.
Grilli, E., F. Menna, and F. Remondino. 2017. “A review of point clouds segmentation and classification algorithms.” In Proc., Int. Archives of the Photogrammetry, Remote Sensing and Spatial Information Sciences, 339–345. Hannover, Germany: International Society for Photogrammetry and Remote Sensing.
Guldur, B., and J. F. Hajjar. 2016. “Automated classification of detected surface damage from point clouds with supervised learning.” In Vol. 33 of Proc., 34th Int. Symp. on Automation and Robotics in Construction (ISARC), 1–7. Taipei, Taiwan: IAARC Publications.
Guldur, B., Y. Yan, and J. F. Hajjar. 2015. “Condition assessment of bridges using terrestrial laser scanners.” In Proc., Structure Congress, 355–366. Reston, VA: ASCE.
Guldur Erkal, B., and J. F. Hajjar. 2017. “Laser-based surface damage detection and quantification using predicted surface properties.” Autom. Constr. 83 (Nov): 285–302. https://doi.org/10.1016/j.autcon.2017.08.004.
Guo, Y., H. Wang, Q. Hu, H. Liu, L. Liu, and M. Bennamoun. 2020. “Deep learning for 3D point clouds: A survey.” IEEE Trans. Pattern Anal. Mach. Intell. 43 (12): 4338–4364. https://doi.org/10.1109/TPAMI.2020.3005434.
Haurum, J. B., M. M. Allahham, M. S. Lynge, K. S. Henriksen, I. A. Nikolov, and T. B. Moeslund. 2021. “Sewer defect classification using synthetic point clouds.” In Proc., 16th Int. Conf. on Computer Vision Theory and Applications, 891–900. Setúbal, Portugal: SciTePress.
Hou, T.-C., J.-W. Liu, and Y.-W. Liu. 2017. “Algorithmic clustering of LiDAR point cloud data for textural damage identifications of structural elements.” Measurement 108 (Oct): 77–90. https://doi.org/10.1016/j.measurement.2017.05.032.
Jiang, H., F. Yan, J. Cai, J. Zheng, and J. Xiao. 2020. “End-to-end 3D point cloud instance segmentation without detection.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 12796–12805. New York: IEEE.
Kalfarisi, R., Z. Y. Wu, and K. Soh. 2020. “Crack detection and segmentation using deep learning with 3D reality mesh model for quantitative assessment and integrated visualization.” J. Comput. Civ. Eng. 34 (3): 04020010. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000890.
Khaloo, A., D. Lattanzi, K. Cunningham, R. Dell’Andrea, and M. Riley. 2018. “Unmanned aerial vehicle inspection of the Placer River Trail Bridge through image-based 3D modelling.” Struct. Infrastruct. Eng. 14 (1): 124–136. https://doi.org/10.1080/15732479.2017.1330891.
Kim, H., and C. Kim. 2020. “Deep-learning-based classification of point clouds for bridge inspection.” Remote Sens. 12 (22): 3757. https://doi.org/10.3390/rs12223757.
Kim, H., J. Yoon, and S.-H. Sim. 2020. “Automated bridge component recognition from point clouds using deep learning.” Struct. Control. Health Monit. 27 (9): e2591. https://doi.org/10.1002/stc.2591.
Kim, I.-H., H. Jeon, S.-C. Baek, W.-H. Hong, and H.-J. Jung. 2018. “Application of crack identification techniques for an aging concrete bridge inspection using an unmanned aerial vehicle.” Sensors 18 (6): 1881. https://doi.org/10.3390/s18061881.
Kim, M. K., H. Sohn, and C.-C. Chang. 2015. “Localization and quantification of concrete spalling defects using terrestrial laser scanning.” J. Comput. Civ. Eng. 29 (6): 1–12. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000415.
Kim, M.-K., H. Sohn, and C.-C. Chang. 2015. “Localization and quantification of concrete spalling defects using terrestrial laser scanning.” J. Comput. Civ. Eng. 29 (6): 04014086. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000415.
Koch, C., K. Georgieva, V. Kasireddy, B. Akinci, and P. Fieguth. 2015. “A review on computer vision based defect detection and condition assessment of concrete and asphalt civil infrastructure.” Adv. Eng. Inf. 29 (2): 196–210. https://doi.org/10.1016/j.aei.2015.01.008.
Koo, B., R. Jung, Y. Yu, and I. Kim. 2021. “A geometric deep learning approach for checking element-to-entity mappings in infrastructure building information models.” J. Comput. Des. Eng. 8 (1): 239–250. https://doi.org/10.1093/jcde/qwaa075.
Laefer, D. F., L. Truong-Hong, H. Carr, and M. Singh. 2014. “Crack detection limits in unit based masonry with terrestrial laser scanning.” NDT & E Int. 62 (Mar): 66–76. https://doi.org/10.1016/j.ndteint.2013.11.001.
Lee, D., J. Kim, and D. Lee. 2019. “Robust concrete crack detection using deep learning-based semantic segmentation.” Int. J. Aeronaut. Space Sci. 20 (1): 287–299. https://doi.org/10.1007/s42405-018-0120-5.
Li, C.-L., M. Zaheer, Y. Zhang, B. Poczos, and R. Salakhutdinov. 2018. “Point cloud GAN.” Preprint, submitted October 13, 2018. https://doi.org/10.48550/arXiv.1810.05795.
Liu, W., S. Chen, D. Boyajian, and E. Hauser. 2010. “Application of 3D LiDAR scan of a bridge under static load testing.” Mater. Eval. 68 (12): 1359–1367.
Liu, W., S. Chen, and E. Hauser. 2011. “LiDAR-based bridge structure defect detection.” Exp. Tech. 35 (6): 27–34. https://doi.org/10.1111/j.1747-1567.2010.00644.x.
Maguire, M., S. Dorafshan, and R. Thomas. 2018. “SDNET2018: A concrete crack image dataset for machine learning applications.” Accessed May 17, 2018. https://digitalcommons.usu.edu/all_datasets/48/.
Martin-Abadal, M., M. Piñar-Molina, A. Martorell-Torres, G. Oliver-Codina, and Y. Gonzalez-Cid. 2021. “Underwater pipe and valve 3D recognition using deep learning segmentation.” J. Mar. Sci. Eng. 9 (1): 5. https://doi.org/10.3390/jmse9010005.
Maru, M. B., D. Lee, K. D. Tola, and S. Park. 2021. “Comparison of depth camera and terrestrial laser scanner in monitoring structural deflections.” Sensors 21 (1): 201. https://doi.org/10.3390/s21010201.
Maturana, D., and S. Scherer. 2015. “Voxnet: A 3D convolutional neural network for real-time object recognition.” In Proc., IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), 922–928. New York: IEEE.
McLaughlin, E., N. Charron, and S. Narasimhan. 2020. “Automated defect quantification in concrete bridges using robotics and deep learning.” J. Comput. Civ. Eng. 34 (5): 04020029. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000915.
Mizoguchi, T., Y. Koda, I. Iwaki, H. Wakabayashi, Y. Kobayashi, K. Shirai, Y. Hara, and H. S. Lee. 2013. “Quantitative scaling evaluation of concrete structures based on terrestrial laser scanning.” Autom. Constr. 35 (Nov): 263–274. https://doi.org/10.1016/j.autcon.2013.05.022.
Mohammadi, M. E., R. L. Wood, and C. E. Wittich. 2019. “Non-temporal point cloud analysis for surface damage in civil structures.” ISPRS Int. J. Geo-Inf. 8 (12): 527. https://doi.org/10.3390/ijgi8120527.
Mohammed Abdelkader, E., O. Moselhi, M. Marzouk, and T. Zayed. 2021. “Entropy-based automated method for detection and assessment of spalling severities in reinforced concrete bridges.” J. Perform. Constr. Facil. 35 (1): 04020132. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001544.
Montufar, G. F., R. Pascanu, K. Cho, and Y. Bengio. 2014. “On the number of linear regions of deep neural networks.” Adv. Neural Inf. Process. Syst. 27: 1–9. https://doi.org/10.48550/arXiv.1402.1869.
Nasrollahi, M., N. Bolourian, and A. Hammad. 2019. “Concrete surface defect detection using deep neural network based on LiDAR scanning.” In Proc, 7th Int. CSCE Conf., 1–10. Montreal: Canadian Society for Civil Engineering.
Olsen, M. J., F. Kuester, B. J. Chang, and T. C. Hutchinson. 2010. “Terrestrial laser scanning-based structural damage assessment.” J. Comput. Civ. Eng. 24 (3): 264–272. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000028.
Pierdicca, R., M. Paolanti, F. Matrone, M. Martini, C. Morbidoni, E. S. Malinverni, E. Frontoni, and A. M. Lingua. 2020. “Point cloud semantic segmentation using a deep learning framework for cultural heritage.” Remote Sens. 12 (6): 1005. https://doi.org/10.3390/rs12061005.
Qi, C. R., H. Su, K. Mo, and L. J. Guibas. 2017a. “PointNet: Deep learning on point sets for 3Dd classification and segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 4–23. New York: IEEE.
Qi, C. R., L. Yi, H. Su, and L. J. Guibas. 2017b. “PointNet++: Deep hierarchical feature learning on point sets in a metric space.” In Advances in neural information processing systems, 5105–5114. New York: Curran Associates.
Rahman, M. A., and Y. Wang. 2016. “Optimizing intersection-over-union in deep neural networks for image segmentation.” In Proc., Int. Symp. on Visual Computing, 234–244. Cham, Switzerland: Springer.
Rashidi, M., M. Mohammadi, S. Sadeghlou Kivi, M. Abdolvand, L. Truong-Hong, and B. Samali. 2020. “A decade of modern bridge monitoring using terrestrial laser scanning: Review and future directions.” Remote Sens. 12 (22): 3796. https://doi.org/10.3390/rs12223796.
Shijie, J., W. Ping, J. Peiyi, and H. Siping. 2017. “Research on data augmentation for image classification based on convolution neural networks.” In Proc., 2017 Chinese Automation Congress (CAC), 4165–4170. New York: IEEE.
Silverman, R. 2002. Modern calculus and analytic geometry. New York: Dover.
Te, G., W. Hu, A. Zheng, and Z. Guo. 2018. “Rgcnn: Regularized graph CNN for point cloud segmentation.” In Proc., 26th ACM Int. Conf. Multimedia, 746–754. New York: Association for Computing Machinery.
Teza, G., A. Galgaro, and F. Moro. 2009. “Contactless recognition of concrete surface damage from laser scanning and curvature computation.” Non-Destr. Test. Eval. Int. 42 (4): 240–249. https://doi.org/10.1016/j.ndteint.2008.10.009.
Truong-Hong, L., H. Falter, D. Lennon, and D. F. Laefer. 2016. “Framework for bridge inspection with laser scanning.” In Proc., 14th East Asia-Pacific Conf. on Structural Engineering and Construction (EASEC-14), 1–9. Ho Chi Minh City, Vietnam: EASEC.
Turkan, Y., J. Hong, S. Laflamme, and N. Puri. 2018. “Adaptive wavelet neural network for terrestrial laser scanner-based crack detection.” Autom. Constr. 94 (Oct): 191–202. https://doi.org/10.1016/j.autcon.2018.06.017.
Valença, J., I. Puente, E. Júlio, H. González-Jorge, and P. Arias-Sánchez. 2017. “Assessment of cracks on concrete bridges using image processing supported by laser scanning survey.” Constr. Build. Mater. 146 (Aug): 668–678. https://doi.org/10.1016/j.conbuildmat.2017.04.096.
Wang, J., Y. Liu, X. Nie, and Y. L. Mo. 2022. “Deep convolutional neural networks for semantic segmentation of cracks.” Struct. Control Health Monit. 29 (1): e2850. https://doi.org/10.1002/stc.2850.
Wang, Z., and F. Lu. 2016. “VoxSegNet: Volumetric CNNs for semantic part segmentation of 3D shapes.” IEEE Trans. Visual Comput. Graphics 26 (9): 2919–2930. https://doi.org/10.1109/TVCG.2019.2896310.
Wu, Z., S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao. 2015. “3D ShapeNets: A deep representation for volumetric shapes.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 1912–1920. New York: IEEE.
Xia, T., J. Yang, and L. Chen. 2022. “Automated semantic segmentation of bridge point cloud based on local descriptor and machine learning.” Autom. Constr. 133 (Jan): 103992. https://doi.org/10.1016/j.autcon.2021.103992.
Yavartanoo, M., E. Y. Kim, and K. M. Lee. 2018. “SPNet: Deep 3D object classification and retrieval using stereographic projection.” In Proc., Asian Conf. on Computer Vision, 691–706. Berlin: Springer.
Yin, C., B. Wang, V. J. L. Gan, M. Wang, and J. C. P. Cheng. 2021. “Automated semantic segmentation of industrial point clouds using ResPointNet++.” Autom. Constr. 130 (Oct): 103874. https://doi.org/10.1016/j.autcon.2021.103874.
Zhang, W., Z. Zhang, D. Qi, and Y. Liu. 2014. “Automatic crack detection and classification method for subway tunnel safety monitoring.” Sensors 14 (10): 19307–19328. https://doi.org/10.3390/s141019307.
Zhao, R., M. Pang, and J. Wang. 2018. “Classifying airborne LiDAR point clouds via deep features learned by a multi-scale convolutional neural network.” Int. J. Geogr. Inf. Sci. 32 (5): 960–979. https://doi.org/10.1080/13658816.2018.1431840.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 2 • March 2023
Copyright
© 2022 American Society of Civil Engineers.
History
Received: May 13, 2022
Accepted: Sep 13, 2022
Published online: Nov 21, 2022
Published in print: Mar 1, 2023
Discussion open until: Apr 21, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Junyan Ma, Xin Jiang, Duan Zheng, Xiaoping Liao, Juan Lu, Yunlong Zhao, TransWallNet: High-Performance Semantic Segmentation of Large-Scale and Multifeatured Point Clouds of Building Gables, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-14827, 150, 8, (2024).
- Patrick Herbers, Firdes Çelik, Markus König, Autonomous Defect Inspection and Mapping for Building Maintenance, Computing in Civil Engineering 2023, 10.1061/9780784485248.124, (1039-1046), (2024).
- Joram Ntiyakunze, Tomo Inoue, Segmentation of Structural Elements from 3D Point Cloud Using Spatial Dependencies for Sustainability Studies, Sensors, 10.3390/s23041924, 23, 4, (1924), (2023).