Building and Infrastructure Defect Detection and Visualization Using Drone and Deep Learning Technologies
Publication: Journal of Performance of Constructed Facilities
Volume 35, Issue 6
Abstract
This paper presents an accurate and stable method for object and defect detection and visualization on building and infrastructural facilities. This method uses drones and cameras to collect three-dimensional (3D) point clouds via photogrammetry, and uses orthographic or arbitrary views of the target objects to generate the feature images of points’ spectral, elevation, and normal features. U-Net is implemented in the pixelwise segmentation for object and defect detection using multiple feature images. This method was validated on four applications, including on-site path detection, pavement cracking detection, highway slope detection, and building facade window detection. The comparative experimental results confirmed that U-Net with multiple features has a better pixelwise segmentation performance than separately using each single feature. The developed method can implement object and defect detection with different shapes, including striped objects, thin objects, recurring and regularly shaped objects, and bulky objects, which will improve the accuracy and efficiency of inspection, assessment, and management of buildings and infrastructural facilities.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The model training and testing data sets are available from the corresponding author upon request. The Python codes are also available from the corresponding author upon request.
Acknowledgments
This work was financially supported by the McShane Endowment fund at Marquette University. The authors are thankful for the reviewers’ valuable comments.
References
Ali, L., N. K. Valappil, D. N. A. Kareem, M. J. John, and H. Al Jassmi. 2019. “Pavement crack detection and localization using convolutional neural networks (CNNs).” In Proc., 2019 Int. Conf. on Digitization (ICD), 217–221. New York: IEEE.
Alipour, M., D. K. Harris, and G. R. Miller. 2019. “Robust pixel-level crack detection using deep fully convolutional neural networks.” J. Comput. Civ. Eng. 33 (6): 04019040. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000854.
Augustaukas, R., and A. Lipnickas. 2019. “Pixel-wise road pavement defects detection using U-net deep neural network.” In Proc., 2019 10th IEEE Int. Conf. on Intelligent Data Acquisition and Advanced Computing Systems: Technology and Applications (IDAACS), 468–471. New York: IEEE.
Autodesk. 2020. “3D view.” Accessed July 4, 2020. https://help.autodesk.com/view/RECAP/ENU/?guid=Reality_Capture_View_and_Navigate_Point_Cloud_Color_Settings_3D_View_html.
Badrinarayanan, V., A. Kendall, and R. Cipolla. 2017. “SegNet: A deep convolutional encoder-decoder architecture for image segmentation.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (12): 2481–2495. https://doi.org/10.1109/TPAMI.2016.2644615.
Chen, L. C., Y. Zhu, G. Papandreou, F. Schroff, and H. Adam. 2018. “Encoder-decoder with atrous separable convolution for semantic image segmentation.” In Proc., Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 833–851. Cham, Switzerland: Springer.
Chollet, F. 2020. “Concatenate layer.” Accessed June 13, 2020. https://keras.io/api/layers/merging_layers/concatenate/.
Dadrasjavan, F., N. Zarrinpanjeh, A. Ameri, G. Engineering, and Q. Branch. 2019. “Automatic crack detection of road pavement based on aerial UAV imagery.” Preprints 2019: 2019070009. https://doi.org/10.20944/preprints201907.0009.v1.
Dorafshan, S., R. J. Thomas, and M. Maguire. 2019. “Benchmarking image processing algorithms for unmanned aerial system-assisted crack detection in concrete structures.” Infrastructures 4 (2): 19. https://doi.org/10.3390/infrastructures4020019.
Dung, C. V., and L. D. Anh. 2019. “Autonomous concrete crack detection using deep fully convolutional neural network.” Autom. Constr. 99 (Mar): 52–58. https://doi.org/10.1016/j.autcon.2018.11.028.
Edmondson, V., J. Woodward, M. Lim, M. Kane, J. Martin, and I. Shyha. 2019. “Improved non-contact 3D field and processing techniques to achieve macrotexture characterisation of pavements.” Constr. Build. Mater. 227 (Dec): 116693. https://doi.org/10.1016/j.conbuildmat.2019.116693.
Fan, R., M. J. Bocus, Y. Zhu, J. Jiao, L. Wang, F. Ma, S. Cheng, and M. Liu. 2019. “Road crack detection using deep convolutional neural network and adaptive thresholding.” In Proc., 2019 IEEE Intelligent Vehicles Symp. (IV), 474–479. New York: IEEE.
Hsieh, Y.-A., and Y. J. Tsai. 2020. “Machine learning for crack detection: Review and model performance comparison.” J. Comput. Civ. Eng. 34 (5): 04020038. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000918.
Intel. 2020. “Intel RealSense LiDAR camera L515.” Accessed August 24, 2020. https://www.intelrealsense.com/lidar-camera-l515/.
Ji, A., X. Xue, Y. Wang, X. Luo, and W. Xue. 2020. “An integrated approach to automatic pixel-level crack detection and quantification of asphalt pavement.” Autom. Constr. 114 (Jun): 103176. https://doi.org/10.1016/j.autcon.2020.103176.
Jiang, Y. 2020a. “As-built CAD drawing tool.” Accessed November 9, 2020. https://www.yuhanjiang.com/research/DT/CAD.
Jiang, Y. 2020b. “Data augmentation.” Accessed November 9, 2020. https://www.yuhanjiang.com/research/FM/DA.
Jiang, Y. 2020c. “Object detection via point cloud and U-net.” Accessed November 9, 2020. https://www.yuhanjiang.com/research/FM/PC.
Jiang, Y., and Y. Bai. 2020. “Estimation of construction site elevations using drone-based orthoimagery and deep learning.” J. Constr. Eng. Manage. 146 (8): 04020086. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001869.
Jiang, Y., Y. Bai, and S. Han. 2020. “Determining ground elevations covered by vegetation on construction sites using drone-based orthoimage and convolutional neural network.” J. Comput. Civ. Eng. 34 (6): 04020049. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000930.
Jiang, Y., S. Han, and Y. Bai. 2021. “Development of a pavement evaluation tool using aerial imagery and deep learning.” J. Transp. Eng. Part B Pavements 147 (3): 04021027. https://doi.org/10.1061/JPEODX.0000282.
Kalfarisi, R., Z. Y. Wu, and K. Soh. 2020. “Crack detection and segmentation using deep learning with 3D reality mesh model for quantitative assessment and integrated visualization.” J. Comput. Civ. Eng. 34 (3): 04020010. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000890.
Kearney, S. P., N. C. Coops, S. Sethi, and G. B. Stenhouse. 2020. “Maintaining accurate, current, rural road network data: An extraction and updating routine using RapidEye, participatory GIS and deep learning.” Int. J. Appl. Earth Obs. Geoinf. 87 (May): 102031. https://doi.org/10.1016/j.jag.2019.102031.
Li, Z., C. Cheng, M. P. Kwan, X. Tong, and S. Tian. 2019. “Identifying asphalt pavement distress using UAV LiDAR point cloud data and random forest classification.” ISPRS Int. J. Geo-Inf. 8 (1): 39. https://doi.org/10.3390/ijgi8010039.
Liu, Y., J. K. W. Yeoh, and D. K. H. Chua. 2020. “Deep learning–based enhancement of motion blurred UAV concrete crack images.” J. Comput. Civ. Eng. 34 (5): 04020028. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000907.
Liu, Z., Y. Cao, Y. Wang, and W. Wang. 2019. “Computer vision-based concrete crack detection using U-net fully convolutional networks.” Autom. Constr. 104 (Aug): 129–139. https://doi.org/10.1016/j.autcon.2019.04.005.
Majidifard, H., Y. Adu-Gyamfi, and W. G. Buttlar. 2020. “Deep machine learning approach to develop a new asphalt pavement condition index.” Constr. Build. Mater. 247 (Jun): 118513. https://doi.org/10.1016/j.conbuildmat.2020.118513.
Maniat, M. 2019. “Deep learning-based visual crack detection using Google Street View images.” Ph.D. thesis, Dept. of Civil Engineering, Univ. of Memphis.
McLaughlin, E., N. Charron, and S. Narasimhan. 2020. “Automated defect quantification in concrete bridges using robotics and deep learning.” J. Comput. Civ. Eng. 34 (5): 04020029. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000915.
Noh, H., S. Hong, and B. Han. 2015. “Learning deconvolution network for semantic segmentation.” In Proc., 2015 IEEE Int. Conf. on Computer Vision (ICCV), 1520–1528. New York: IEEE.
OpenCV. 2020. “Contours in OpenCV.” Accessed November 9, 2020. https://docs.opencv.org/3.4/d3/d05/tutorial_py_table_of_contents_contours.html.
Pix4D. 2018a. “Highway.” Accessed July 31, 2020. https://cloud.pix4d.com/dataset/272051/map?shareToken=6fb4d298cade4a8c81b206f68fd757c7.
Pix4D. 2018b. “Michigan central station.” Accessed July 31, 2020. https://cloud.pix4d.com/dataset/258513/files/inputs?shareToken=060367e6115f4185902cd33556a70e38.
Pix4D. 2018c. “Quarry 2.0.” Accessed July 31, 2020. https://cloud.pix4d.com/dataset/256164/map?shareToken=97a07d231fbc47b1b105d6cc7bcab0a4.
Propeller Aero. 2018. “What is ground sample distance (GSD) and how does it affect your drone data?” Accessed June 13, 2020. https://www.propelleraero.com/blog/ground-sample-distance-gsd-calculate-drone-data/.
Protopapadakis, E., A. Voulodimos, A. Doulamis, N. Doulamis, and T. Stathaki. 2019. “Automatic crack detection for tunnel inspection using deep learning and heuristic image post-processing.” Appl. Intell. 49 (7): 2793–2806. https://doi.org/10.1007/s10489-018-01396-y.
Roberts, R., L. Inzerillo, and G. Di Mino. 2020. “Exploiting low-cost 3D imagery for the purposes of detecting and analyzing pavement distresses.” Infrastructures 5 (1): 6. https://doi.org/10.3390/infrastructures5010006.
Ronneberger, O., P. Fischer, and T. Brox. 2015. “U-net: Convolutional networks for biomedical image segmentation.” In Proc., Lecture Notes in Computer Science (Including Subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 234–241. Cham, Switzerland: Springer.
Shang, Z., J. Bradley, and Z. Shen. 2020. “A co-optimal coverage path planning method for aerial scanning of complex structures.” Expert Syst. Appl. 158 (Nov): 113535. https://doi.org/10.1016/j.eswa.2020.113535.
Shang, Z., and Z. Shen. 2018. “Real-time 3D reconstruction on construction site using visual SLAM and UAV.” In Proc., Construction Research Congress 2018, 305–315. Reston, VA: ASCE.
Shang, Z., and Z. Shen. 2019. “Indoor testing and simulation platform for close-distance visual inspection of complex structures using micro quadrotor UAV.” Preprint, submitted April 10, 2019. https://arxiv.org/abs/1904.05271.
Shelhamer, E., J. Long, and T. Darrell. 2017. “Fully convolutional networks for semantic segmentation.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (4): 640–651.
Shi, Y., L. Cui, Z. Qi, F. Meng, and Z. Chen. 2016. “Automatic road crack detection using random structured forests.” IEEE Trans. Intell. Transp. Syst. 17 (12): 3434–3445. https://doi.org/10.1109/TITS.2016.2552248.
Song, W., G. Jia, H. Zhu, D. Jia, and L. Gao. 2020. “Automated pavement crack damage detection using deep multiscale convolutional features.” J. Adv. Transp. 2020 (Jan): 1–11. https://doi.org/10.1155/2020/6412562.
Stacks, D. L. 2019. “Pavement manual: Visual pavement condition surveys.” Accessed May 1, 2020. http://onlinemanuals.txdot.gov/txdotmanuals/pdm/visual_p_cond_surveys.htm.
Yang, Q., W. Shi, J. Chen, and W. Lin. 2020. “Deep convolution neural network-based transfer learning method for civil infrastructure crack detection.” Autom. Constr. 116 (Aug): 103199. https://doi.org/10.1016/j.autcon.2020.103199.
Yu, F., and V. Koltun. 2015. “Multi-scale context aggregation by dilated convolutions.” Preprint, submitted November 23, 2015. https://arxiv.org/abs/1511.07122.
Zhang, A., K. C. P. Wang, Y. Fei, Y. Liu, C. Chen, G. Yang, J. Q. Li, E. Yang, and S. Qiu. 2019. “Automated pixel-level pavement crack detection on 3D asphalt surfaces with a recurrent neural network.” Comput.-Aided Civ. Infrastruct. Eng. 34 (3): 213–229. https://doi.org/10.1111/mice.12409.
Zhang, K., Y. Zhang, and H.-D. Cheng. 2020. “CrackGAN: Pavement crack detection using partially accurate ground truths based on generative adversarial learning.” IEEE Trans. Intell. Transp. Syst. 22 (2): 1306–1319. https://doi.org/10.1109/TITS.2020.2990703.
Zhao, H., J. Shi, X. Qi, X. Wang, and J. Jia. 2017. “Pyramid scene parsing network.” In Proc., 2017 IEEE Conf. on Computer Vision and Pattern Recognition (CVPR), 6230–6239. New York: IEEE. https://ieeexplore.ieee.org/document/8100143.
Zhou, S., and W. Song. 2020a. “Deep learning-based roadway crack classification using laser-scanned range images: A comparative study on hyperparameter selection.” Autom. Constr. 114 (Jun): 103171. https://doi.org/10.1016/j.autcon.2020.103171.
Zhou, S., and W. Song. 2020b. “Robust image-based surface crack detection using range data.” J. Comput. Civ. Eng. 34 (2): 04019054. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000873.
Zou, Q., Z. Zhang, Q. Li, X. Qi, Q. Wang, and S. Wang. 2019. “DeepCrack: Learning hierarchical convolutional features for crack detection.” IEEE Trans. Image Process. 28 (3): 1498–1512. https://doi.org/10.1109/TIP.2018.2878966.
Information & Authors
Information
Published In
Journal of Performance of Constructed Facilities
Volume 35 • Issue 6 • December 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Aug 8, 2020
Accepted: Jul 2, 2021
Published online: Sep 25, 2021
Published in print: Dec 1, 2021
Discussion open until: Feb 25, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Hyun-Jung Woo, Won-Hwa Hong, Jintak Oh, Seung-Chan Baek, Defining Structural Cracks in Exterior Walls of Concrete Buildings Using an Unmanned Aerial Vehicle, Drones, 10.3390/drones7030149, 7, 3, (149), (2023).
- Sisi Han, Yuhan Jiang, Yilei Huang, Mingzhu Wang, Yong Bai, Andrea Spool-White, Scan2Drawing: Use of Deep Learning for As-Built Model Landscape Architecture, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-13077, 149, 5, (2023).
- Weiming Chen, Changfan Li, Hailin Guo, A lightweight face-assisted object detection model for welding helmet use, Expert Systems with Applications, 10.1016/j.eswa.2023.119764, 221, (119764), (2023).
- Michael Y. L. Chew, Vincent J. L. Gan, Long-Standing Themes and Future Prospects for the Inspection and Maintenance of Façade Falling Objects from Tall Buildings, Sensors, 10.3390/s22166070, 22, 16, (6070), (2022).
- Yuliang Zhao, Jian Li, Weishi Li, Peng Shan, Xiaoai Wang, Lianjiang Li, Qiang Fu, MS-IAF: Multi-Scale Information Augmentation Framework for Aircraft Detection, Remote Sensing, 10.3390/rs14153696, 14, 15, (3696), (2022).
- Bartosz Ptak, Dominik Pieczyński, Mateusz Piechocki, Marek Kraft, On-Board Crowd Counting and Density Estimation Using Low Altitude Unmanned Aerial Vehicles—Looking beyond Beating the Benchmark, Remote Sensing, 10.3390/rs14102288, 14, 10, (2288), (2022).
- Yuhan Jiang, Yilei Huang, Jingkuang Liu, Dapeng Li, Shuiyuan Li, Weijing Nie, In-Hun Chung, Automatic Volume Calculation and Mapping of Construction and Demolition Debris Using Drones, Deep Learning, and GIS, Drones, 10.3390/drones6100279, 6, 10, (279), (2022).
- Luqman Ali, Fady Alnajjar, Wasif Khan, Mohamed Adel Serhani, Hamad Al Jassmi, Bibliometric Analysis and Review of Deep Learning-Based Crack Detection Literature Published between 2010 and 2022, Buildings, 10.3390/buildings12040432, 12, 4, (432), (2022).
- Yuhan Jiang, Sisi Han, Yong Bai, Scan4Façade: Automated As-Is Façade Modeling of Historic High-Rise Buildings Using Drones and AI, Journal of Architectural Engineering, 10.1061/(ASCE)AE.1943-5568.0000564, 28, 4, (2022).
- Yuhan Jiang, Sisi Han, Dapeng Li, Yong Bai, Mingzhu Wang, Automatic concrete sidewalk deficiency detection and mapping with deep learning, Expert Systems with Applications, 10.1016/j.eswa.2022.117980, 207, (117980), (2022).