Encoding 3D Point Contexts for Self-Supervised Spall Classification Using 3D Bridge Point Clouds
Publication: Journal of Computing in Civil Engineering
Volume 37, Issue 2
Abstract
Geometric features, such as normal and curvature, have been prominently used for point cloud-based unsupervised spall classification. In addition, some researchers use hand-crafted features (e.g., out-of-plane distance, eccentricity, principal curvatures in 2D slices). These features perform well in low noise settings; however, the performance tapers down significantly when the quality of point clouds is affected by factors such as higher noise and inconsistent point-to-point spacing. Instead of relying purely on handcrafted features, the research presented in this paper investigates the potential for combining domain knowledge with deep learning to automatically learn better quality defect-sensitive features for point cloud-based spall classification. Specifically, generic three dimensional (3D) shape and 3D neighborhood features have been encoded as inputs to a deep autoencoder network for self-supervised spall classification from point clouds. Overall, this approach only resulted in marginal improvement over classification results from the current state-of-the-art unsupervised approach that uses handcrafted features. However, significant improvement in the results were observed in datasets that had higher noise levels. Given that noise is pervasive in datasets from outdoor settings like civil infrastructure, this added robustness to noise improves the reliability of point cloud-based condition assessment for concrete bridges.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data that support the findings of this study are available from the corresponding author upon reasonable request, particularly the quasi-real point cloud data generated by the authors in the form of point cloud files (in.ply format) and corresponding ground truth (in.mat format).
Acknowledgments
The project is funded by a grant from the National Science Foundation (NSF), #1328930. The authors gratefully acknowledge NSF’s support. The authors also acknowledge the kind support from project collaborators Dr. Scherer and Luke Yoder for sharing the data collected by UAV during field studies, and Michael Baker International and Pennsylvania Department of Transportation for giving permission to scan a bridge. Any opinions, findings, conclusions or recommendations presented in this paper are those of authors and do not necessarily reflect the views of the NSF and other patrons.
References
AASHTO. 2013. Manual for bridge element inspection. Washington, DC: AASHTO.
Aljalbout, E., V. Golkov, Y. Siddiqui, M. Strobel, and D. Cremers. 2018. “Clustering with deep learning: Taxonomy and new methods.” Preprint, submitted January 23, 2018. https://arxiv.org/abs/1801.07648.
Aljemely, A. H., J. Xuan, F. K. J. Jawad, O. Al-Azzawi, and A. S. Alhumaima. 2020. “A novel unsupervised learning method for intelligent fault diagnosis of rolling element bearings based on deep functional auto-encoder.” J. Mech. Sci. Technol. 34 (11): 4367–4381. https://doi.org/10.1007/s12206-020-1002-x.
Armeni, I., O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese. 2016. “3D semantic parsing of large-scale indoor spaces.” In Proc., IEEE Computation Social Conf. Computer Vision Pattern Recognition, 1534–1543. New York: IEEE.
Beckman, G. H., D. Polyzois, and Y. J. Cha. 2019. “Deep learning-based automatic volumetric damage quantification using depth camera.” Autom. Constr. 99 (Mar): 114–124. https://doi.org/10.1016/j.autcon.2018.12.006.
Bhatt, P. M., R. K. Malhan, P. Rajendran, B. C. Shah, S. Thakar, Y. J. Yoon, and S. K. Gupta. 2021. “Image-Based surface defect detection using deep learning: A review.” J. Comput. Inf. Sci. Eng. 21 (4): 040801.
Blomley, R., M. Weinmann, J. Leitloff, and B. Jutzi. 2014. “Shape distribution features for point cloud analysis—A geometric histogram approach on multiple scales.” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 2 (3): 9–16. https://doi.org/10.5194/isprsannals-ii-3-9-2014.
Cabaleiro, M., B. Riveiro, B. Conde, and A. Sánchez-Rodríguez. 2020. “A case study of measurements of deformations due to different loads in pieces less than 1 m from lidar data.” Meas. J. Int. Meas. Confed. 151 (Feb): 107196. https://doi.org/10.1016/j.measurement.2019.107196.
Cha, Y. J., and Z. Wang. 2018. “Unsupervised novelty detection–based structural damage localization using a density peaks-based fast clustering algorithm.” Struct. Health Monit. 17 (2): 313–324. https://doi.org/10.1177/1475921717691260.
Chen, S., L. Truong-Hong, D. F. Laefer, and E. Mangina. 2018. “Automated bridge deck evaluation through UAV derived point cloud.” In Proc., 2018 Civil Engineering Research in Ireland Conf., 735–740. Lille, France: Civil Engineering Research Association of Ireland.
Chow, J. K., Z. Su, J. Wu, P. S. Tan, X. Mao, and Y. H. Wang. 2020. “Anomaly detection of defects on concrete structures with the convolutional autoencoder.” Adv. Eng. Inf. 45 (Aug): 101105. https://doi.org/10.1016/j.aei.2020.101105.
Czimmermann, T., G. Ciuti, M. Milazzo, M. Chiurazzi, S. Roccella, C. M. Oddo, and P. Dario. 2020. “Visual-based defect detection and classification approaches for industrial applications—A survey.” Sensors 20 (5): 1459. https://doi.org/10.3390/s20051459.
Diez, A., N. L. D. Khoa, M. Makki Alamdari, Y. Wang, F. Chen, and P. Runcie. 2016. “A clustering approach for structural health monitoring on bridges.” J. Civ. Struct. Health Monit. 6 (3): 429–445. https://doi.org/10.1007/s13349-016-0160-0.
Ding, Q., W. Chen, B. King, Y. Liu, and G. Liu. 2013. “Combination of overlap-driven adjustment and Phong model for LiDAR intensity correction.” ISPRS J. Photogramm. Remote Sens. 75 (Jan): 40–47. https://doi.org/10.1016/j.isprsjprs.2012.09.015.
Dung, C. V., and L. D. Anh. 2019. “Autonomous concrete crack detection using deep fully convolutional neural network.” Autom. Constr. 99 (Mar): 52–58. https://doi.org/10.1016/j.autcon.2018.11.028.
Entezami, A., H. Shariatmadar, and A. Karamodin. 2019. “Data-driven damage diagnosis under environmental and operational variability by novel statistical pattern recognition methods.” Struct. Health Monit. 18 (5–6): 1416–1443. https://doi.org/10.1177/1475921718800306.
FARO. 2016. “Faro X330 specification sheet.” Accessed September 27, 2022. https://www.laserscanning-europe.com/en/system/files/04REF201-610-EN-FARO_LaserScannerFocus3DX330HDRTechSheet.pdf.
Farrar, C. R., and K. Worden. 2012. Structural health monitoring: A machine learning perspective. Hoboken, NJ: John Wiley and Sons.
Figueiredo, E., G. Park, C. R. Farrar, K. Worden, and J. Figueiras. 2011. “Machine learning algorithms for damage detection under operational and environmental variability.” Struct. Health Monit. 10 (6): 559–572. https://doi.org/10.1177/1475921710388971.
García-Macías, E., and F. Ubertini. 2022. “Integrated SHM systems: Damage detection through unsupervised learning and data fusion.” Struct. Integr. 2022 (1): 247–268. https://doi.org/10.1007/978-3-030-81716-9_12.
Guldur, B., and J. F. Hajjar. 2016. “Automated classification of detected surface damage from point clouds with supervised learning.” In Proc., 33rd Int. Symp. Association for Automation and Robotics in Construction, 307–313. Oulu, Finland: International Association for Automation and Robotics in Construction.
Guldur Erkal, B., and J. F. Hajjar. 2017. “Laser-based surface damage detection and quantification using predicted surface properties.” Autom. Constr. 83 (Nov): 285–302. https://doi.org/10.1016/j.autcon.2017.08.004.
Hackel, T., N. Savinov, L. Ladicky, J. D. Wegner, K. Schindler, and M. Pollefeys. 2017. “Semantic3d.Net: A new large-scale point cloud classification benchmark.” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 2017 (1): 91–98. https://doi.org/10.5194/isprs-annals-IV-1-W1-91-2017.
Hassani, K., and M. Haley. 2019. “Unsupervised multi-task feature learning on point clouds.” In Proc., IEEE Int. Conf. Computation Visualities, 8159–8170. Piscataway, NJ: IEEE Signal Processing Society.
Hershey, J. R., Z. Chen, J. Le Roux, and S. Watanabe. 2016. “Deep clustering: Discriminative embeddings for segmentation and separation.” In Proc., ICASSP, IEEE Int. Conf. Acoustician Speech Signal Process, 31–35. Piscataway, NJ: IEEE Signal Processing Society.
Hsu, P. H., and Z. Y. Zhuang. 2020. “Incorporating handcrafted features into deep learning for point cloud classification.” Remote Sens. 12 (22): 3713. https://doi.org/10.3390/rs12223713.
Hüthwohl, P., and I. Brilakis. 2018. “Detecting healthy concrete surfaces.” Adv. Eng. Inf. 37 (Aug): 150–162. https://doi.org/10.1016/j.aei.2018.05.004.
Isailović, D., V. Stojanovic, M. Trapp, R. Richter, R. Hajdin, and J. Döllner. 2020. “Bridge damage: Detection, IFC-based semantic enrichment and visualization.” Autom. Constr. 112 (Apr): 103088. https://doi.org/10.1016/J.AUTCON.2020.103088.
Jaritz, M., T. H. Vu, R. de Charette, E. Wirbel, and P. Pérez. 2020. “xMUDA: Cross-Modal unsupervised domain adaptation for 3D semantic segmentation.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 12602–12611. New York: IEEE.
Kang, D., S. S. Benipal, D. L. Gopal, and Y. J. Cha. 2020. “Hybrid pixel-level concrete crack segmentation and quantification across complex backgrounds using deep learning.” Autom. Constr. 118 (Oct): 103291. https://doi.org/10.1016/j.autcon.2020.103291.
Kashani, A., M. Olsen, C. Parrish, and N. Wilson. 2015. “A review of LIDAR radiometric processing: From ad hoc intensity correction to rigorous radiometric calibration.” Sensors 15 (11): 28099–28128. https://doi.org/10.3390/s151128099.
Kasireddy, V. 2021. Point cloud-based unsupervised spall classification in bridges. Pittsburgh, PA: Carnegie Mellon Univ.
Kasireddy, V., and B. Akinci. 2022. “Assessing the impact of 3D point neighborhood size selection on unsupervised spall classification with 3D bridge point clouds.” Adv. Eng. Inf. 52 (Apr): 101624. https://doi.org/10.1016/j.aei.2022.101624.
Kim, M.-K., H. Sohn, and C.-C. Chang. 2015. “Localization and quantification of concrete spalling defects using terrestrial laser scanning.” J. Comput. Civ. Eng. 29 (6): 04014086. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000415.
Kingma, D. P., and J. L. Ba. 2015. “Adam: A method for stochastic optimization.” In Proc., 3rd Int. Conf. for Learning Representations. La Jolla, CA: International Conference on Learning Representations.
Li, X., L. Zhang, and Z. Zhu. 2021. “SnapshotNet: Self-supervised feature learning for point cloud data segmentation using minimal labeled data.” Comput. Vision Image Understanding 216 (Feb): 103339. https://doi.org/10.1016/j.cviu.2021.103339.
Lin, P., and J. Tao. 2019. “A novel bearing health indicator construction method based on ensemble stacked autoencoder.” In Proc., IEEE Int. Conf. on Prognostics and Health Management. New York: IEEE.
Liu, Z., Y. Cao, Y. Wang, and W. Wang. 2019. “Computer vision-based concrete crack detection using U-net fully convolutional networks.” Autom. Constr. 104 (Aug): 129–139. https://doi.org/10.1016/j.autcon.2019.04.005.
Lu, R., I. Brilakis, and C. R. Middleton. 2019. “Detection of structural components in point clouds of existing RC bridges.” Comput. Civ. Infrastruct. Eng. 34 (3): 191–212. https://doi.org/10.1111/mice.12407.
McLaughlin, E., N. Charron, and S. Narasimhan. 2020. “Automated defect quantification in concrete bridges using robotics and deep learning.” J. Comput. Civ. Eng. 34 (5): 04020029. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000915.
Mohammadi, M. E. 2019. “Point cloud analysis for surface defects in civil structures.” Ph.D. thesis, Dept. of Civil Engineering, Univ. of Nebraska-Lincoln.
Mohammadi, M. E., and R. L. Wood. 2022. Machine learning-based structural damage identification within three-dimensional point clouds. Berlin: Springer.
Mohammadi, M. E., R. L. Wood, and C. E. Wittich. 2019. “Non-temporal point cloud analysis for surface damage in civil structures.” ISPRS Int. J. Geo-Inf. 8 (12): 527. https://doi.org/10.3390/ijgi8120527.
Mohammed Abdelkader, E., O. Moselhi, M. Marzouk, and T. Zayed. 2021. “Entropy-based automated method for detection and assessment of spalling severities in reinforced concrete bridges.” J. Perform. Constr. Facil. 35 (1): 04020132. https://doi.org/10.1061/(ASCE)CF.1943-5509.0001544.
Nasrollahi, M., N. Bolourian, and A. Hammad. 2019. “Concrete surface defect detection using deep neural network based on LiDAR scanning.” In Proc., 7th Int. Construcion Conf. Jointly With the Construction Research Congress. Reston, VA: ASCE.
Neeli, Y. S. 2020. “Use of photogrammetry aided damage detection for residual strength estimation of corrosion damaged prestressed concrete bridge girders.” Ph.D. thesis, Master of Science in Civil Engineering, Virginia Technology.
NJDOT. 2014. Structural evaluation and bridge management. Ewing Township, NJ: NJDOT.
Olsen, M. J., F. Kuester, B. J. Chang, and T. C. Hutchinson. 2010. “Terrestrial laser scanning-based structural damage assessment.” J. Comput. Civ. Eng. 24 (3): 264–272. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000028.
Omar, T., and M. L. Nehdi. 2018. “Condition assessment of reinforced concrete bridges: Current practice and research challenges.” Infrastructures 3 (3): 36. https://doi.org/10.3390/infrastructures3030036.
Osada, R., T. Funkhouser, B. Chazelle, and D. Dobkin. 2002. “Shape distributions.” ACM Trans. Graph. 21 (4): 807–832. https://doi.org/10.1145/571647.571648.
Rafiei, M. H., and H. Adeli. 2018. “A novel unsupervised deep learning model for global and local health condition assessment of structures.” Eng. Struct. 156 (Feb): 598–607. https://doi.org/10.1016/j.engstruct.2017.10.070.
Rao, Y., J. Lu, and J. Zhou. 2020. “Global-local bidirectional reasoning for unsupervised representation learning of 3D point clouds.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition, 5375–5384. Washington, DC: IEEE Computer Society.
Robinson, B. M., W. English, D. Langford, J. Walters, B. Angotta, S. Fowler, and R. Diltz. 2018. “Scanning LiDAR for airfield damage assessment.” In Vol. 10646 of Proc., SPIE. Bellingham, WA: SPIE.
Rumelhart, D. E., G. E. Hinton, and R. J. Williams. 2013. “Learning internal representations by error propagation.” Readings Cogn. Sci. A Perspect. Psychol. Artif. Intell. 2013 (1): 399–421.
Sarmadi, H., and A. Karamodin. 2020. “A novel anomaly detection method based on adaptive Mahalanobis-squared distance and one-class kNN rule for structural health monitoring under environmental effects.” Mech. Syst. Sig. Process. 140 (Jun): 106495. https://doi.org/10.1016/j.ymssp.2019.106495.
Shafiei dizaji, M., and D. Harris. 2019. “3D InspectionNet: A deep 3D convolutional neural networks based approach for 3D defect detection on concrete columns.” In Vol. 10971 of Proc., SPIE. Bellingham, WA: SPIE.
Suchocki, C., M. Jagoda, R. Obuchovski, D. Šlikas, and J. Sužiedelye-Visockiene. 2018. “The properties of terrestrial laser system intensity in measurements of technical conditions of architectural structures.” Metrol. Meas. Syst. Index 25 (4): 779–792. https://doi.org/10.24425/mms.2018.124886.
Tan, K., and X. Cheng. 2015. “Intensity data correction based on incidence angle and distance for terrestrial laser scanner.” J. Appl. Remote Sens. 9 (1): 094094. https://doi.org/10.1117/1.JRS.9.094094.
Torok, M. M., M. Golparvar-Fard, and K. B. Kochersberger. 2014. “Image-based automated 3D crack detection for post-disaster building assessment.” J. Comput. Civ. Eng. 28 (5): A4014004. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000334.
Wang, L., K. Kawaguchi, and P. Wang. 2020. “Damaged ceiling detection and localization in large-span structures using convolutional neural networks.” Autom. Constr. 116 (Aug): 103230. https://doi.org/10.1016/j.autcon.2020.103230.
Wang, N., X. Zhao, P. Zhao, Y. Zhang, Z. Zou, and J. Ou. 2019. “Automatic damage detection of historic masonry buildings based on mobile deep learning.” Autom. Constr. 103 (Jul): 53–66. https://doi.org/10.1016/j.autcon.2019.03.003.
Wang, Z., and Y. J. Cha. 2021. “Unsupervised deep learning approach using a deep auto-encoder with a one-class support vector machine to detect damage.” Struct. Health Monit. 20 (1): 406–425. https://doi.org/10.1177/1475921720934051.
Weinmann, M. 2016. 3D scene analysis, 141–224. Berlin: Springer.
Weinmann, M., B. Jutzi, S. Hinz, and C. Mallet. 2015. “Semantic point cloud interpretation based on optimal neighborhoods, relevant features and efficient classifiers.” ISPRS J. Photogramm. Remote Sens. 105 (Jul): 286–304. https://doi.org/10.1016/j.isprsjprs.2015.01.016.
Weinmann, M., B. Jutzi, and C. Mallet. 2014. “Semantic 3D scene interpretation: A framework combining optimal neighborhood size selection with relevant features.” ISPRS Ann. Photogramm. Remote Sens. Spatial Inf. Sci. 2 (3): 181–188. https://doi.org/10.5194/isprsannals-ii-3-181-2014.
Wu, H., X. Ao, Z. Chen, C. Liu, Z. Xu, P. Yu, and S. H. Hong. 2019. “Concrete Spalling Detection for Metro Tunnel from Point Cloud Based on Roughness Descriptor.” J. Sensors 2019 (May): 8574750. https://doi.org/10.1155/2019/8574750.
Yang, B., X. Fu, N. D. Sidiropoulos, and M. Hong. 2017. “Towards K-means-friendly spaces: Simultaneous deep learning and clustering.” In Proc., Int. Conf. on Machine Learning, 5888–5901. San Diego: International Machine Learning Society.
Yang, J., S. Li, Z. Wang, H. Dong, J. Wang, and S. Tang. 2020. “Using deep learning to detect defects in manufacturing: A comprehensive survey and current challenges.” Materials (Basel). 13 (24): 1–23. https://doi.org/10.3390/ma13245755.
Ye, J., and N. Toyama. 2021. “Benchmarking deep learning models for automatic ultrasonic imaging inspection.” IEEE Access 9 (Mar): 36986–36994. https://doi.org/10.1109/ACCESS.2021.3062860.
Ye, X. W., T. Jin, and C. B. Yun. 2019. “A review on deep learning-based structural health monitoring of civil infrastructures.” Smart Struct. Syst. 24 (5): 567–585. https://doi.org/10.12989/sss.2019.24.5.567.
Zhang, D., X. Xu, H. Lin, R. Gui, M. Cao, and L. He. 2019. “Automatic road-marking detection and measurement from laser-scanning 3D profile data.” Autom. Constr. 108 (Dec): 102957. https://doi.org/10.1016/j.autcon.2019.102957.
Zhang, J., X. Hu, and H. Dai. 2021. “Unsupervised learning of ALS point clouds for 3-D terrain scene clustering.” In Proc., IEEE Geoscience and Remote Sensing Letters. New York: IEEE. https://doi.org/10.1109/LGRS.2020.3047096.
Zhang, L., and Z. Zhu. 2019. “Unsupervised feature learning for point cloud understanding by contrasting and clustering using graph convolutional neural networks.” In Proc., Int. Conf. on 3D Vision (3DV), 395–404. Piscataway, NJ: IEEE.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 37 • Issue 2 • March 2023
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Jun 6, 2022
Accepted: Oct 25, 2022
Published online: Dec 29, 2022
Published in print: Mar 1, 2023
Discussion open until: May 29, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.