Deep Learning–Based Autonomous Road Condition Assessment Leveraging Inexpensive RGB and Depth Sensors and Heterogeneous Data Fusion: Pothole Detection and Quantification
Publication: Journal of Transportation Engineering, Part B: Pavements
Volume 149, Issue 2
Abstract
Poor condition of roads is a major factor for traffic accidents and damage to vehicles. A significant portion of car accidents is attributed to severe three-dimensional (3D) pavement distresses such as potholes, ruttings, and ravelings. Insufficient road condition assessment is responsible for the poor condition of roads. To inspect the condition of the pavement surfaces more frequently and efficiently, an inexpensive data acquisition system was developed that consists of a consumer-grade RGB-D sensor and an edge computing device that can be mounted on vehicles and collect data while driving vehicles. The RGB-D sensor is used for collecting two-dimensional (2D) color images and corresponding 3D depth data, and the lightweight edge computing device is used to control the RGB-D sensor and store the collected data. An RGB-D pavement surface data set is generated. Furthermore, encoder-decoder deep convolutional neural networks (DCNNs) consisting of one or two encoders, and one decoder trained on heterogeneous RGB-D pavement surface data are used for pothole segmentation. Comprehensive experiments using different depth encoding techniques and data fusion methods including data- and feature-level fusion were performed to investigate the efficacy of defect detection using DCNNs. Experimental results demonstrate that the feature-level RGB-D data fusion based on the surface normal encoding of depth data outperform other approaches in terms of segmentation accuracy, where the mean intersection over union (IoU) over 10-fold cross-validation of 0.82 is achieved that shows a 7.7% improvement compared with a network trained only on RGB data. In addition, this study explores the efficacy of indirectly using depth information for pothole detection when depth data are not available. Additionally, the semantic segmentation results were utilized to quantify the severity level of the potholes assisting in maintenance decision-making. The result from these comprehensive experiments using an RGB-D pavement surface data set gathered through the proposed data acquisition system is a stepping stone for opportunistic data collection and processing through crowdsourcing and Internet of Things in future smart cities for effective road assessment. Finally, suggestions about the improvement of the proposed system are discussed.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
The authors would like to acknowledge Zhao Xing Lim, Da Cheng, and Xianmeng Zhang from the Elmore Family School of Electrical and Computer Engineering at Purdue University for their support during the development of the data acquisition collection system.
References
Akagic, A., E. Buza, and S. Omanovic. 2017. “Pothole detection: An efficient vision based method using RGB color space image segmentation.” In Proc., 2017 40th Int. Convention on Information and Communication Technology, Electronics and Microelectronics (MIPRO), 1104–1109. New York: IEEE.
Alhashim, I., and P. Wonka. 2018. “High quality monocular depth estimation via transfer learning.” Preprint, submitted December 31, 2018. http://arxiv.org/abs/1812.11941.
ASCE. 2021. “2021 report card for America’s infrastructure.” Accessed March 1, 2022. https://infrastructurereportcard.org/.
Barbarella, M., M. R. De Blasiis, and M. Fiani. 2019. “Terrestrial laser scanner for the analysis of airport pavement geometry.” Int. J. Pavement Eng. 20 (4): 466–480. https://doi.org/10.1080/10298436.2017.1309194.
Botshekan, M., et al. 2020. “Roughness-induced vehicle energy dissipation from crowdsourced smartphone measurements through random vibration theory.” Data-Centric Eng. 1 (Dec): e16. https://doi.org/10.1017/dce.2020.17.
Botshekan, M., E. Asaadi, J. Roxon, F.-J. Ulm, M. Tootkaboni, and A. Louhghalam. 2021. “Smartphone-enabled road condition monitoring: From accelerations to road roughness and excess energy dissipation.” Proc. R. Soc. A 477 (2246): 20200701. https://doi.org/10.1098/rspa.2020.0701.
Cao, M.-T., Q.-V. Tran, N.-M. Nguyen, and K.-T. Chang. 2020. “Survey on performance of deep learning models for detecting road damages using multiple dashcam image resources.” Adv. Eng. Inf. 46 (Oct): 101182. https://doi.org/10.1016/j.aei.2020.101182.
Chang, A., A. Dai, T. Funkhouser, T. Halber, M. Niessner, M. Savva, S. Song, A. Zeng, and Y. Zhang. 2017. “Matterport3D: Learning from RGB-D data in indoor environments.” Preprint, submitted September 18, 2017. http://arxiv.org/abs/1709.06158.
Chen, F.-C., and M. R. Jahanshahi. 2017. “NB-CNN: Deep learning-based crack detection using convolutional neural network and naïve bayes data fusion.” IEEE Trans. Ind. Electron. 65 (5): 4392–4400. https://doi.org/10.1109/TIE.2017.2764844.
Chen, X., Q. Dong, H. Zhu, and B. Huang. 2016a. “Development of distress condition index of asphalt pavements using ltpp data through structural equation modeling.” Transp. Res. Part C Emerging Technol. 68 (Jul): 58–69. https://doi.org/10.1016/j.trc.2016.03.011.
Chen, Y. L., M. R. Jahanshahi, P. Manjunatha, W. Gan, M. Abdelbarr, S. F. Masri, B. Becerik-Gerber, and J. P. Caffrey. 2016b. “Inexpensive multimodal sensor fusion system for autonomous data acquisition of road surface conditions.” IEEE Sens. J. 16 (21): 7731–7743. https://doi.org/10.1109/JSEN.2016.2602871.
Chun, C., and S.-K. Ryu. 2019. “Road surface damage detection using fully convolutional neural networks and semi-supervised learning.” Sensors 19 (24): 5501. https://doi.org/10.3390/s19245501.
Deng, J., W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. 2009. “Imagenet: A large-scale hierarchical image database.” In Proc., 2009 IEEE Conf. on Computer Vision and Pattern Recognition, 248–255. New York: IEEE.
Dung, C. V. 2019. “Autonomous concrete crack detection using deep fully convolutional neural network.” Autom. Constr. 99 (Mar): 52–58. https://doi.org/10.1016/j.autcon.2018.11.028.
Fan, R., U. Ozgunalp, B. Hosking, M. Liu, and I. Pitas. 2019. “Pothole detection based on disparity transformation and road surface modeling.” IEEE Trans. Image Process. 29 (Aug): 897–908. https://doi.org/10.1109/TIP.2019.2933750.
Fan, R., H. Wang, Y. Wang, M. Liu, and I. Pitas. 2021. “Graph attention layer evolves semantic segmentation for road pothole detection: A benchmark and algorithms.” IEEE Trans. Image Process. 30 (Sep): 8144–8154. https://doi.org/10.1109/TIP.2021.3112316.
Fischler, M. A., and R. C. Bolles. 1981. “Random sample consensus: A paradigm for model fitting with applications to image analysis and automated cartography.” Commun. ACM 24 (6): 381–395. https://doi.org/10.1145/358669.358692.
Girshick, R. 2015. “Fast R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 1440–1448. New York: IEEE.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 580–587. New York: IEEE.
Gupta, S., R. Girshick, P. Arbeláez, and J. Malik. 2014. “Learning rich features from RGB-D images for object detection and segmentation.” In Proc., European Conf. on Computer Vision, 345–360. Cham, Switzerland: Springer.
Hazirbas, C., L. Ma, C. Domokos, and D. Cremers. 2016. “FuseNet: Incorporating depth into semantic segmentation via fusion-based CNN architecture.” In Proc., Asian Conf. on Computer Vision, 213–228. Cham, Switzerland: Springer.
Hoffman, J., S. Gupta, and T. Darrell. 2016. “Learning with side information through modality hallucination.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 826–834. New York: IEEE.
Huang, G., Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. 2017. “Densely connected convolutional networks.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 4700–4708. New York: IEEE.
Ibragimov, E., H.-J. Lee, J.-J. Lee, and N. Kim. 2022. “Automated pavement distress detection using region based convolutional neural networks.” Int. J. Pavement Eng. 23 (6): 1981–1992. https://doi.org/10.1080/10298436.2020.1833204.
Jahanshahi, M. R., F. Jazizadeh, S. F. Masri, and B. Becerik-Gerber. 2013. “Unsupervised approach for autonomous pavement-defect detection and quantification using an inexpensive depth sensor.” J. Comput. Civ. Eng. 27 (6): 743–754. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000245.
Jo, Y., and S. Ryu. 2015. “Pothole detection system using a black-box camera.” Sensors 15 (11): 29316–29331. https://doi.org/10.3390/s151129316.
Kim, J.-H. 2009. “Estimating classification error rate: Repeated cross-validation, repeated hold-out and bootstrap.” Comput. Stat. Data Anal. 53 (11): 3735–3745. https://doi.org/10.1016/j.csda.2009.04.009.
Kingma, D. P., and J. Ba. 2014. “Adam: A method for stochastic optimization.” Preprint, submitted December 22, 2014. http://arxiv.org/abs/1412.6980.
LeCun, Y., Y. Bengio, and G. Hinton. 2015. “Deep learning.” Nature 521 (7553): 436–444. https://doi.org/10.1038/nature14539.
Li, Z., Y. Gan, X. Liang, Y. Yu, H. Cheng, and L. Lin. 2016. “LSTM-CF: Unifying context modeling and fusion with LSTMS for RGB-D scene labeling.” In Proc., European Conf. on Computer Vision, 541–557. Cham, Switzerland: Springer.
Maeda, H., Y. Sekimoto, T. Seto, T. Kashiyama, and H. Omata. 2018. “Road damage detection and classification using deep neural networks with smartphone images.” Comput.-Aided Civ. Infrastruct. Eng. 33 (12): 1127–1141. https://doi.org/10.1111/mice.12387.
Mahmoudzadeh, A., A. Golroo, M. R. Jahanshahi, and S. Firoozi Yeganeh. 2019. “Estimating pavement roughness by fusing color and depth data obtained from an inexpensive RGB-D sensor.” Sensors 19 (7): 1655. https://doi.org/10.3390/s19071655.
Miller, J. S., and W. Y. Bellinger. 2003. Distress identification manual for the long-term pavement performance program. McLean, VA: Federal Highway Administration, Office of Infrastructure.
Moazzam, I., K. Kamal, S. Mathavan, S. Usman, and M. Rahman. 2013. “Metrology and visualization of potholes using the Microsoft Kinect sensor.” In Proc., 16th Int. IEEE Conf. on Intelligent Transportation Systems (ITSC 2013), 1284–1291. New York: IEEE.
NVIDIA. 2016. “NVIDIA jetson solutions for drones and UAVs.” Accessed February 14, 2023. https://developer.nvidia.com/embedded/community/quick-start-platforms.
Paszke, A., S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. 2017. “Automatic differentiation in PyTorch.” In Proc., 31st Conf. on Neural Information Processing Systems. Red Hook, NY: Curran Associates.
Piryonesi, S. M., and T. El-Diraby. 2021. “Climate change impact on infrastructure: A machine learning solution for predicting pavement condition index.” Constr. Build. Mater. 306 (Nov): 124905. https://doi.org/10.1016/j.conbuildmat.2021.124905.
Quigley, M., K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng. 2009. “ROS: An open-source robot operating system.” In Vol. 3 of Proc., ICRA Workshop on Open Source Software, 5. New York: IEEE.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” IEEE Trans. Pattern Anal. Mach. Intell. 39 (6): 1137–1149. https://doi.org/10.1109/TPAMI.2016.2577031.
Silberman, N., D. Hoiem, P. Kohli, and R. Fergus. 2012. “Indoor segmentation and support inference from RGBD images.” In Vol. 7576 of Proc., Computer Vision—ECCV 2012. Lecture Notes in Computer Science, edited by A. Fitzgibbon, S. Lazebnik, P. Perona, Y. Sato, and C. Schmid. Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-642-33715-4_54.
Simonyan, K., and A. Zisserman. 2014. “Very deep convolutional networks for large-scale image recognition.” Preprint, submitted September 4, 2014. http://arxiv.org/abs/1409.1556.
Song, S., S. P. Lichtenberg, and J. Xiao. 2015. “Sun RGB-D: A RGB-D scene understanding benchmark suite.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 567–576. New York: IEEE.
TRIP. 2015. “The interstate highway system turns 60: Challenges to its ability to continue to save lives, time and money.” Accessed March 1, 2022. https://infrastructureusa.org/the-interstate-highway-system-turns-60-challenges-to-its-ability-to-continue-to-save-lives-time-and-money/.
Tsai, Y.-C., and A. Chatterjee. 2018. “Pothole detection and classification using 3D technology and watershed method.” J. Comput. Civ. Eng. 32 (2): 04017078. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000726.
Ukhwah, E. N., E. M. Yuniarno, and Y. K. Suprapto. 2019. “Asphalt pavement pothole detection using deep learning method based on YOLO neural network.” In Proc., 2019 Int. Seminar on Intelligent Technology and Its Applications (ISITIA), 35–40. New York: IEEE.
Wang, A., J. Cai, J. Lu, and T.-J. Cham. 2015. “MMSS: Multi-modal sharable and specific feature learning for RGB-D object recognition.” In Proc., IEEE Int. Conf. on Computer Vision, 1125–1133. New York: IEEE.
Wu, R.-T., A. Singla, M. R. Jahanshahi, E. Bertino, B. J. Ko, and D. Verma. 2019. “Pruning deep convolutional neural networks for efficient edge computing in condition assessment of infrastructures.” Comput.-Aided Civ. Infrastruct. Eng. 34 (9): 774–789. https://doi.org/10.1111/mice.12449.
Zhang, A., K. C. Wang, B. Li, E. Yang, X. Dai, Y. Peng, Y. Fei, Y. Liu, J. Q. Li, and C. Chen. 2017. “Automated pixel-level pavement crack detection on 3d asphalt surfaces using a deep-learning network.” Comput.-Aided Civ. Infrastruct. Eng. 32 (10): 805–819. https://doi.org/10.1111/mice.12297.
Zhang, C., and A. Elaksher. 2012. “An unmanned aerial vehicle-based imaging system for 3D measurement of unpaved road surface distresses.” Comput.-Aided Civ. Infrastruct. Eng. 27 (2): 118–129. https://doi.org/10.1111/j.1467-8667.2011.00727.x.
Zhang, D., Q. Zou, H. Lin, X. Xu, L. He, R. Gui, and Q. Li. 2018. “Automatic pavement defect detection using 3D laser profiling technology.” Autom. Constr. 96 (Dec): 350–365. https://doi.org/10.1016/j.autcon.2018.09.019.
Zhang, L., F. Yang, Y. D. Zhang, and Y. J. Zhu. 2016. “Road crack detection using deep convolutional neural network.” In Proc., 2016 IEEE Int. Conf. on Image Processing (ICIP), 3708–3712. New York: IEEE.
Zhou, S., and W. Song. 2021. “Crack segmentation through deep convolutional neural networks and heterogeneous image fusion.” Autom. Constr. 125 (May): 103605. https://doi.org/10.1016/j.autcon.2021.103605.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part B: Pavements
Volume 149 • Issue 2 • June 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Mar 29, 2022
Accepted: Dec 24, 2022
Published online: Mar 10, 2023
Published in print: Jun 1, 2023
Discussion open until: Aug 10, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Li Li, Dandan Liu, Li Teng, Jie Zhu, Development of a Relationship between Pavement Condition Index and Riding Quality Index on Rural Roads: A Case Study in China, Mathematics, 10.3390/math12030410, 12, 3, (410), (2024).
- Jinguk Kim, Woohoon Jeon, Seoungbum Kim, Evaluating Road Hazard Maintenance Efficiency Using Citizen Science Data to Improve Road Safety, Sustainability, 10.3390/su152015074, 15, 20, (15074), (2023).
- Vallamdas Ratheesh, S.Govinda Rao, V. Keerthi, A Comparative CNN based Analysis of Classification and Detection of Potholes, 2023 7th International Conference on Electronics, Communication and Aerospace Technology (ICECA), 10.1109/ICECA58529.2023.10395532, (611-619), (2023).
- A. Mounika Yesaswini, K. Annapurna, A Hybrid Approach for Intrusion Detection System to Enhance Feature Selection, 2023 Second International Conference on Augmented Intelligence and Sustainable Systems (ICAISS), 10.1109/ICAISS58487.2023.10250515, (1301-1307), (2023).