Development of a Car-Free Street Mapping Model Using an Integrated System with Unmanned Aerial Vehicles, Aerial Mapping Cameras, and a Deep Learning Algorithm
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 3
Abstract
Road condition and quality are critical road maintenance and risk reduction factors. Most existing road monitoring systems include regular on-site surveys and maintenance. However, major roads in urban areas are generally complicated and have heavy traffic during the daytime, so such field investigation can be significantly limited. Moreover, any road work at nighttime can be risky and dangerous and incur excessive expenses. Based on a review of existing systems for monitoring road conditions, this study focuses on overcoming two unsolved challenges: the capacity of the monitoring range and the avoidance techniques to ensure traffic is not hindered. To solve these challenges, these paper proposes an integrated road monitoring system called Car-free Street Mapping (CfSM) using unmanned aerial vehicles (UAV), aerial mapping cameras, and deep learning (DL) algorithms. The use of the aerial mapping camera mounted on the UAV is to widen the monitoring viewing range, and general-purpose drones are used in this study rather than expensive special equipment. Since the drone-taken images include many passing vehicles that conceal the road surface from the camera vision, the DL model was applied to detect the vehicles and their shadows and then remove them from the images. To train the DL model, two image datasets were used: publicly available cars overhead with context (COWC) images and orthoimages additionally taken for the project to further improve the accuracy. The two datasets consist of 298,623 labeled objects on 9,331 images in total. The tests resulted in a mean average precision (mAP) of 89.57% for trucks, 95.77% for passenger vehicles, and 76.51% for buses. Finally, the object-removed images were composited into one whole car-free image. The CfSM was applied to two areas in Yeouido and Sangam-dong, Seoul, Korea. The car-free images in both regions show a spatial resolution of 10 mm and can be used for various purposes such as road maintenance and management and autonomous vehicle roadmaps.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This study was conducted with the Seoul Innovation Challenge 2017 (IC170011) and the Public Testbed for Technology Innovation 2018 (IU180005), which are projects of Seoul Metropolitan City and Seoul Business Agency (SBA), respectively. This paper is also a part of the project “Innovation in Construction Automation & Technologies” by the Australian Government through the Department of Foreign Affairs and Trade.
References
Abbondati, F., S. A. Biancardo, R. Veropalumbo, and G. Dell’Acqua. 2021. “Surface monitoring of road pavements using mobile crowdsensing technology.” Measurement 171 (Feb): 108763. https://doi.org/10.1016/j.measurement.2020.108763.
Aggarwal, C. C. 2018. Outlier analysis. New York: Springer.
Alessandroni, G., L. Klopfenstein, S. Delpriori, M. Dromedari, G. Luchetti, B. Paolini, A. Seraghiti, E. Lattanzi, V. Freschi, and A. Carini. 2014. “Smartroadsense: Collaborative road surface condition monitoring.” In Proc., UBICOMM: 210–215. Wilmington, DE: International Academy, Research, and Industry Association.
Alfarrarjeh, A., D. Trivedi, S. H. Kim, and C. Shahabi. 2018. “A deep learning approach for road damage detection from smartphone images.” In Proc., 2018 IEEE Int. Conf. on Big Data (Big Data). New York: IEEE.
Alqudah, Y. A., and B. H. Sababha. 2017. On the analysis of road surface conditions using embedded smartphone sensors.” In Proc., 2017 8th Int. Conf. on Information and Communication Systems (ICICS). New York: IEEE.
Bhatt, P., S. Gupta, P. Singh, and P. Dhiman. 2017. “Accident and road quality assessment using android Google maps API.” In Proc., 2017 Int. Conf. on Computing, Communication and Automation (ICCCA). New York: IEEE.
Chang, J.-R., H.-M. Hsu, and S.-J. Chao. 2012. “Development of a road monitoring and reporting system based on location-based services and augmented-reality technologies.” J. Perform. Constr. Facil. 26 (6): 812–823. https://doi.org/10.1061/(ASCE)CF.1943-5509.0000272.
Chen, K., M. Lu, G. Tan, and J. Wu. 2013. “CRSM: Crowdsourcing based road surface monitoring.” In Proc., 2013 IEEE 10th Int. Conf. on High Performance Computing and Communications & 2013 IEEE Int. Conf. on Embedded and Ubiquitous Computing. New York: IEEE.
Chowdhury, A., G. Karmakar, J. Kamruzzaman, and S. Islam. 2020. “Trustworthiness of self-driving vehicles for intelligent transportation systems in industry applications.” IEEE Trans. Ind. Inf. 17 (2): 961–970. https://doi.org/10.1109/TII.2020.2987431.
Chuang, T.-Y., N.-H. Perng, and J.-Y. Han. 2019. “Pavement performance monitoring and anomaly recognition based on crowdsourcing spatiotemporal data.” Autom. Constr. 106 (Oct): 102882. https://doi.org/10.1016/j.autcon.2019.102882.
El-Wakeel, A. S., A. Osman, A. Noureldin, and H. S. Hassanein. 2017. “Road test experiments and statistical analysis for real-time monitoring of road surface conditions.” In Proc., GLOBECOM 2017-2017 IEEE Global Communications Conf. New York: IEEE.
Eriksson, J., L. Girod, B. Hull, R. Newton, S. Madden, and H. Balakrishnan. 2008. “The pothole patrol: Using a mobile sensor network for road surface monitoring.” In Proc., 6th Int. Conf. on Mobile Systems, Applications, and Services. New York: Association for Computing Machinery.
Etten, A. V., 2019. “Satellite imagery multiscale rapid detection with windowed networks.” In Proc., 2019 IEEE Winter Conf. on Applications of Computer Vision (WACV), 735–743. New York: IEEE.
Gong, L., L. An, M. Liu, and J. Zhang. 2012. “Road damage detection from high-resolution RS image.” In Proc., 2012 IEEE Int. Geoscience and Remote Sensing Symp. New York: IEEE.
Harikrishnan, P., and V. P. Gopi. 2017. “Vehicle vibration signal processing for road surface monitoring.” IEEE Sens. J. 17 (16): 5192–5197. https://doi.org/10.1109/JSEN.2017.2719865.
He, K., X. Zhang, S. Ren, and J. Sun. 2016. “Deep residual learning for image recognition.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 770–778. New York: IEEE.
Katsuma, R., and S. Yoshida. 2018. “Dynamic routing for emergency vehicle by collecting real-time road conditions.” Int. J. Commun. Network Syst. Sci. 11 (2): 27–44. https://doi.org/10.4236/ijcns.2018.112003.
Kumar, R., A. Mukherjee, and V. Singh. 2017. “Community sensor network for monitoring road roughness using smartphones.” J. Comput. Civ. Eng. 31 (3): 04016059. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000624.
Li, J., Q. Qin, H. Ma, and W. Yuan. 2010. Study on road damage assessment based on RS and GIS.” In Proc., 2010 IEEE Int. Geoscience and Remote Sensing Symp. New York: IEEE.
Li, P., H. Xu, and B. Song. 2011. “A novel method for urban road damage detection using very high resolution satellite imagery and road map.” Photogramm. Eng. Remote Sens. 77 (10): 1057–1066. https://doi.org/10.14358/PERS.77.10.1057.
Lin, T.-Y., P. Goyal, R. Girshick, K. He, and P. Dollár. 2017. Focal loss for dense object detection.” In Proc., IEEE Int. Conf. on Computer Vision, 2980–2988. New York: IEEE.
Maeda, H., Y. Sekimoto, T. Seto, T. Kashiyama, and H. Omata. 2018. “Road damage detection and classification using deep neural networks with smartphone images.” Computer-Aided Civ. Infrastruct. Eng. 33 (12): 1127–1141. https://doi.org/10.1111/mice.12387.
Malin, F., I. Norros, and S. Innamaa. 2019. “Accident risk of road and weather conditions on different road types.” Accid. Anal. Prev. 122 (Jan): 181–188. https://doi.org/10.1016/j.aap.2018.10.014.
Mertz, C. 2011. “Continuous road damage detection using regular service vehicles.” In Proc., ITS World Congress. Washington, DC: ITS America.
Mohan, P., V. N. Padmanabhan, and R. Ramjee. 2008. Nericell: Rich monitoring of road and traffic conditions using mobile smartphones.” In Proc., 6th ACM Conf. on Embedded Network Sensor Systems. New York: Association for Computing Machinery.
Moon, S., N. Ham, S. Kim, L. Hou, J.-H. Kim, and J.-J. Kim. 2020. “Fourth industrialization-oriented offsite construction: Case study of an application to an irregular commercial building.” Eng. Constr. Architect. Manage. 27 (9): 2271–2286.
Mundhenk, T. N., G. Konjevod, W. A. Sakla, and K. Boakye. 2016. “A large contextual dataset for classification, detection and counting of cars with deep learning.” In Proc., European Conf. on Computer Vision. Amsterdam, Netherlands: Springer.
Park, J., K. Im, and J. H. Lee. 2019. “PotholeEye—How can we effectively maintain the pavement distress?” In Proc., 17th Annual Int. Conf. on Mobile Systems, Applications, and Services. New York: Association for Computing Machinery.
Saleh, S. M., S. Sugiarto, A. Hilal, and D. Ariansyah. 2017. “A study on the traffic impact of the road corridors due to flyover construction at Surabaya intersection.” In Proc., AIP Conf. College Park, MD: AIP Publishing LLC.
Sammut, C., and G. I. Webb. 2011. Encyclopedia of machine learning. New York: Springer.
Singh, R. K., and S. Suman. 2012. “Accident analysis and prediction of model on national highways.” Int. J. Adv. Technol. Civ. Eng. 1 (2): 25–30.
Strazdins, G., A. Mednis, G. Kanonirs, R. Zviedris, and L. Selavo. 2011. “Towards vehicular sensor networks with android smartphones for road surface monitoring.” In Proc., 2nd Int. Workshop on Networks of Cooperating Objects (CONET’11), Electronic Proc. CPS Week. Riga, Latvia: Institute of Electronics and Computer Science.
Suda, Y., K. Nakano, H. Sugiyama, R. Hayashi, and S. Yamabe. 2010. “Estimation of road surface conditions from tire vibration.” In Proc., ASME Int. Mechanical Engineering Congress and Exposition. New York: ASME.
SZ DJI Technology Co., Ltd. 2022a. “DJI Phantom 4 Pro - Product Information.” Accessed January 2022. https://www.dji.com/au/phantom-4-pro/info#specs.
SZ DJI Technology Co., Ltd. 2022b. “DJI 9 Mavic 2 Pro - Product Information.” Accessed January 2022. https://www.dji.com/au/mavic-2/info#specs.
Tang, T., J. Li, H. Huang, and X. Yang. 2014. “A car-following model with real-time road conditions and numerical tests.” Measurement 48 (Feb): 63–76. https://doi.org/10.1016/j.measurement.2013.10.035.
Varona, B., A. Monteserin, and A. Teyseyre. 2019. “A deep learning approach to automatic road surface monitoring and pothole detection.” Personal Ubiquitous Comput. 24: 519–534. https://doi.org/10.1007/s00779-019-01234-z.
Vittorio, A., V. Rosolino, I. Teresa, C. M. Vittoria, and P. G. Vincenzo. 2014. “Automated sensing system for monitoring of road surface quality by mobile devices.” Procedia-Social Behav. Sci. 111 (Feb): 242–251. https://doi.org/10.1016/j.sbspro.2014.01.057.
Wang, J., Q. Qin, J. Zhao, X. Ye, X. Qin, X. Yang, J. Wang, X. Zheng, and Y. Sun. 2015. A knowledge-based method for road damage detection using high-resolution remote sensing image.” In Proc., 2015 IEEE Int. Geoscience and Remote Sensing Symp. (IGARSS). New York: IEEE.
Yulianandha Mabrur, A. 2019. “Analisis pemanfaatan opensource dronedeploy dalam proses mozaik foto udara (UAV).” Pawon: Jurnal Arsitektur 3 (2): 79–92. https://doi.org/10.36040/pawon.v3i02.891.
Zekavat, P. R., S. Moon, and L. E. Bernold. 2014. “Performance of short and long range wireless communication technologies in construction.” Autom. Constr. 47 (Nov): 50–61. https://doi.org/10.1016/j.autcon.2014.07.008.
Zekavat, P. R., S. Moon, and L. E. Bernold. 2015. “Holonic construction management: Unified framework for ICT-supported process control.” J. Manage. Eng. 31 (1): A4014008. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000316.
Zhang, C. 2008. “An UAV-based photogrammetric mapping system for road condition assessment.” Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 37: 627–632.
Zhang, C., and A. Elaksher. 2012. “An unmanned aerial vehicle-based imaging system for 3D measurement of unpaved road surface distresses 1.” Comput.-Aided Civ. Infrastruct. Eng. 27 (2): 118–129. https://doi.org/10.1111/j.1467-8667.2011.00727.x.
Zhang, C., N. R. Sabar, E. Chung, A. Bhaskar, and X. Guo. 2019. “Optimisation of lane-changing advisory at the motorway lane drop bottleneck.” Transp. Res. Part C: Emerging Technol. 106 (Sep): 303–316. https://doi.org/10.1016/j.trc.2019.07.016.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 3 • May 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Aug 3, 2021
Accepted: Nov 19, 2021
Published online: Feb 7, 2022
Published in print: May 1, 2022
Discussion open until: Jul 7, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Linjun Lu, Fei Dai, Digitalization of Traffic Scenes in Support of Intelligent Transportation Applications, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5204, 37, 5, (2023).
- Mulan Qiu, Liang Huang, Bo-Hui Tang, ASFF-YOLOv5: Multielement Detection Method for Road Traffic in UAV Images Based on Multiscale Feature Fusion, Remote Sensing, 10.3390/rs14143498, 14, 14, (3498), (2022).
- Chongrui Tian, Fengbin Zhang, Zhaoxiang Li, Ruidong Wang, Xunhua Huang, Liang Xi, Yi Zhang, Intrusion Detection Method Based on Deep Learning, Wireless Communications and Mobile Computing, 10.1155/2022/1338392, 2022, (1-8), (2022).
- Jisoo Park, Yong K. Cho, Sungjin Kim, Deep learning-based UAV image segmentation and inpainting for generating vehicle-free orthomosaic, International Journal of Applied Earth Observation and Geoinformation, 10.1016/j.jag.2022.103111, 115, (103111), (2022).