Abstract
Project control operations in construction are mostly executed via direct observations and the manual monitoring of progress and performance of construction tasks on the job site. Project engineers move physically within job-site areas to ensure activities are executed as planned. Such physical displacements are error-prone and ineffective in cost and time, particularly in larger construction zones. It is critical to explore new methods and technologies to effectively assist performance control operations by rapidly capturing data from materials and equipment on the job site. Motivated by the ubiquitous use of unmanned aerial vehicles (UAVs) in construction projects and the maturity of computer-vision-based machine-learning (ML) techniques, this research investigates the challenges of object detection—the process of predicting classes of objects (specified construction materials and equipment)—in real time. The study addresses the challenges of data collection and predictions for remote monitoring in project control activities. It uses these two proven and robust technologies by exploring factors that impact the use of UAV aerial images to design and implement object detectors through an analytical conceptualization and a showcase demonstration. The approach sheds light on the applications of deep-learning techniques to access and rapidly identify and classify resources in real-time. It paves the way to shift from costly and time-consuming job-site walkthroughs that are coupled with manual data processing and input to more automated, streamlined operations. The research found that the critical factor to develop object detectors with acceptable levels of accuracy is collecting aerial images with for adequate scales with high frequencies from different positions of the same construction areas.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request (SSD trained model, images used for training, and construction project UAV videos).
Acknowledgments
This research was partially supported by the National Science Foundation (NSF) Grant No. IIS 1550833.
References
Albeaino, G., M. Gheisari, and B. W. Franz. 2019. “A systematic review of unmanned aerial vehicle application areas and technologies in the AEC domain.” J. Inf. Technol. Constr. 24: 381–405.
Arabi, S., A. Haghighat, and A. Sharma. 2020. “A deep-learning-based computer vision solution for construction vehicle detection.” Comput.-Aided Civ. Infrastruct. Eng. 35 (7): 753–767. https://doi.org/10.1111/mice.12530.
Asadi, K., A. Kalkunte Suresh, A. Ender, S. Gotad, S. Maniyar, S. Anand, M. Noghabaei, K. Han, E. Lobaton, and T. Wu. 2020. “An integrated UGV-UAV system for construction site data collection.” Autom. Constr. 112 (Apr): 103068. https://doi.org/10.1016/j.autcon.2019.103068.
Avola, D., L. Cinque, G. L. Foresti, N. Martinel, D. Pannone, and C. Piciarelli. 2020. “A UAV video dataset for mosaicking and change detection from low-altitude flights.” IEEE Trans. Syst. Man Cybernetics Syst. 50 (6): 2139–2149. https://doi.org/10.1109/TSMC.2018.2804766.
Azar, E. R., S. Dickinson, and B. McCabe. 2013. “Server-customer interaction tracker: Computer vision-based system to estimate dirt-loading cycles.” J. Constr. Eng. Manage. 139 (7): 785–794. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000652.
Bakirman, T., B. Bayram, B. Akpinar, M. F. Karabulut, O. C. Bayrak, A. Yigitoglu, and D. Z. Seker. 2020. “Implementation of ultra-light UAV systems for cultural heritage documentation.” J. Cult. Heritage 44 (Jul–Aug): 174–184. https://doi.org/10.1016/j.culher.2020.01.006.
Barekatain, M., M. Martí, H. Shih, S. Murray, K. Nakayama, Y. Matsuo, and H. Prendinger. 2017. “Okutama-Action: An aerial view video dataset for concurrent human action detection.” In Proc., 2017 IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW). New York: IEEE.
Brilakis, I., M.-W. Park, and G. Jog. 2011. “Automated vision tracking of project related entities.” Adv. Eng. Inf. 25 (4): 713–724. https://doi.org/10.1016/j.aei.2011.01.003.
Deng, J., W. Dong, R. Socher, L. Li, L. Kai, and F.-F. Li. 2009. “ImageNet: A large-scale hierarchical image database.” In Proc., 2009 IEEE Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Deng, L. 2014. “Deep learning: Methods and applications.” Found. Trends Signal Process. 7 (3–4): 197–387. https://doi.org/10.1561/2000000039.
Dozzi, S. P., and S. M. AbouRizk. 1993. Productivity in construction. Ottawa: Institute for Research in Construction, National Research Council Ottawa.
Duque, L., J. Seo, and J. Wacker. 2018. “Bridge deterioration quantification protocol using UAV.” J. Bridge Eng. 23 (10): 04018080. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001289.
Engel, J., J. Sturm, and D. Cremers. 2014. “Scale-aware navigation of a low-cost quadrocopter with a monocular camera.” Rob. Auton. Syst. 62 (11): 1646–1656. https://doi.org/10.1016/j.robot.2014.03.012.
Everingham, M., L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. 2009. “The pascal visual object classes (VOC) challenge.” Int. J. Comput. Vision 88 (2): 303–338. https://doi.org/10.1007/s11263-009-0275-4.
Fang, W., L. Ding, B. Zhong, P. E. D. Love, and H. Luo. 2018. “Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach.” Adv. Eng. Inf. 37 (Aug): 139–149. https://doi.org/10.1016/j.aei.2018.05.003.
Fang, W., P. E. D. Love, H. Luo, and L. Ding. 2020. “Computer vision for behaviour-based safety in construction: A review and future directions.” Adv. Eng. Inf. 43 (Jan): 100980. https://doi.org/10.1016/j.aei.2019.100980.
Ficapal, A., and I. Mutis. 2019. “Framework for the detection, diagnosis, and evaluation of thermal bridges using infrared thermography and unmanned aerial vehicles.” Buildings 9 (8): 179. https://doi.org/10.3390/buildings9080179.
Gheisari, M., and B. Esmaeili. 2019. “Applications and requirements of unmanned aerial systems (UASs) for construction safety.” Saf. Sci. 118 (Oct): 230–240. https://doi.org/10.1016/j.ssci.2019.05.015.
Girshick, R. 2015. “Fast R-CNN.” In Proc., 2015 IEEE Int. Conf. on Computer Vision (ICCV). New York: IEEE.
Girshick, R., J. Donahue, T. Darrell, and J. Malik. 2014. “Rich feature hierarchies for accurate object detection and semantic segmentation.” In Proc., 2014 IEEE Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Golparvar-Fard, M., A. Heydarian, and J. C. Niebles. 2013. “Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers.” Adv. Eng. Inf. 27 (4): 652–663. https://doi.org/10.1016/j.aei.2013.09.001.
Greenwood, W. W., J. P. Lynch, and D. Zekkos. 2019. “Applications of UAVs in Civil Infrastructure.” J. Infrastruct. Syst. 25 (2): 04019002. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000464.
Irizarry, J., and D. B. Costa. 2016. “Exploratory study of potential applications of unmanned aerial systems for construction management tasks.” J. Manage. Eng. 32 (2): 10. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000422.
Jahr, K., A. Braun, and A. Borrmann. 2018. “Formwork detection in UAV pictures of construction sites.” In Vol. 12 of Proc., eWork and eBusiness in Architecture, Engineering, and Construction, edited by R. S. Jan Karlshoj, 265–271. Copenhagen, Denmark: CRC Press. https://doi.org/10.1201/9780429506215-33.
Jinwoo Kim, S. C. 2015. “Robust real-time object detection on construction sites using integral channel features.” In Proc., Int. Conf. on Construction Engineering and Project Management. Deakin, Australia: International Centre for Complex Project Management.
Kim, D., M. Liu, S. Lee, and V. R. Kamat. 2019. “Remote proximity monitoring between mobile construction resources using camera-mounted UAVs.” Autom. Constr. 99 (Mar): 168–182. https://doi.org/10.1016/j.autcon.2018.12.014.
Kim, H., H. Kim, Y. W. Hong, and H. Byun. 2018. “Detecting construction equipment using a region-based fully convolutional network and transfer learning.” J. Comput. Civ. Eng. 32 (2): 04017082. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000731.
Kim, K., H. Kim, and H. Kim. 2017. “Image-based construction hazard avoidance system using augmented reality in wearable device.” Autom. Constr. 83 (Nov): 390–403. https://doi.org/10.1016/j.autcon.2017.06.014.
Kwon, S., J.-W. Park, D. Moon, S. Jung, and H. Park. 2017. “Smart merging method for hybrid point cloud data using UAV and LIDAR in earthwork construction.” Procedia Eng. 196: 21–28. https://doi.org/10.1016/j.proeng.2017.07.168.
Li, Y., H. Dong, H. Li, X. Zhang, B. Zhang, and Z. Xiao. 2020. “Multi-block SSD based small object detection for UAV railway scene surveillance.” Chin. J. Aeronaut. 33 (6): 1747–1755. https://doi.org/10.1016/j.cja.2020.02.024.
Lin, T., P. Goyal, R. Girshick, K. He, and P. Dollár. 2020. “Focal loss for dense object detection.” IEEE Trans. Pattern Anal. Mach. Intell. 42 (2): 318–327. https://doi.org/10.1109/TPAMI.2018.2858826.
Liu, D., J. Chen, D. Hu, and Z. Zhang. 2019. “Dynamic BIM-augmented UAV safety inspection for water diversion project.” Comput. Ind. 108 (Jun): 163–177. https://doi.org/10.1016/j.compind.2019.03.004.
Liu, W., D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C.-Y. Fu, and A. C. Berg. 2016. “SSD: Single shot multibox detector.” In Proc., European Conf. on Computer Vision, 21–37. Cham, Switzerland: Springer.
Luo, H., C. Xiong, W. Fang, P. E. D. Love, B. Zhang, and X. Ouyang. 2018. “Convolutional neural networks: Computer vision-based workforce activity assessment in construction.” Autom. Constr. 94 (Oct): 282–289. https://doi.org/10.1016/j.autcon.2018.06.007.
Memarzadeh, M., A. Heydarian, M. Golparvar-Fard, and J. C. Niebles. 2012. Real-time and automated recognition and 2D tracking of construction workers and equipment from site video streams.” In Proc., Int. Conf. on Computing in Civil Engineering. Reston, VA: ASCE.
Nath, N. D., T. Chaspari, and A. H. Behzadan. 2019. “Single- and multi-label classification of construction objects using deep transfer learning methods.” J. Inf. Technol. Constr. 24: 511–526. https://doi.org/10.36680/j.itcon.2019.028.
National Research Council. 2009. Advancing the competitiveness and efficiency of the U.S. construction industry. Washington, DC: National Academies Press.
Redmon, J., and A. Farhadi. 2018. “YOLOv3: An incremental improvement.” Preprint, submitted April 8, 2018. http://arxiv.org/abs/1804.02767.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” Preprint, submitted June 4, 2015. http://arxiv.org/abs/1506.01497.
Rezazadeh Azar, E. 2017. “Semantic annotation of videos from equipment-intensive construction operations by shot recognition and probabilistic reasoning.” J. Comput. Civ. Eng. 31 (5): 04017042. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000693.
Roberts, D., and M. Golparvar-Fard. 2019. “End-to-end vision-based detection, tracking, and activity analysis of earthmoving equipment filmed at ground level.” Autom. Constr. 105 (Sep): 102811. https://doi.org/10.1016/j.autcon.2019.04.006.
Seo, J., S. Han, S. Lee, and H. Kim. 2015. “Computer vision techniques for construction safety and health monitoring.” Artif. Intell. Eng. 29 (2): 239–251. https://doi.org/10.1016/j.aei.2015.02.001.
Sherafat, B., C. R. Ahn, R. Akhavian, A. H. Behzadan, M. Golparvar-Fard, H. Kim, Y.-C. Lee, A. Rashidi, and E. R. Azar. 2020. “Automated methods for activity recognition of construction workers and equipment: State-of-the-art review.” J. Constr. Eng. Manage. 146 (6): 03120002. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001843.
Slaton, T., C. Hernandez, and R. Akhavian. 2020. “Construction activity recognition with recurrent convolutional networks.” Autom. Constr. 113 (May): 103138. https://doi.org/10.1016/j.autcon.2020.103138.
Szegedy, C., L. Wei, J. Yangqing, P. Sermanet, S. Reed, D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich. 2015. “Going deeper with convolutions.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Thakar, V., H. Saini, W. Ahmed, M. M. Soltani, A. Aly, and J. Y. Yu. 2018. “Efficient single-shot multibox detector for construction site monitoring.” In Proc., Int. Smart Cities Conf. (ISC2), 1–6. New York: IEEE.
Tzutalin. 2015. “LabelImg.” Accessed January 25, 2020. https://github.com/tzutalin/labelImg.
Wu, B., A. Wan, F. Iandola, P. H. Jin, and K. Keutzer. 2017. “SqueezeDet: Unified, small, low power fully convolutional neural networks for real-time object detection for autonomous driving.” In Proc., 2017 IEEE Conf. on Computer Vision and Pattern Recognition Workshops (CVPRW). New York: IEEE.
Xiang, X., N. Lv, X. Guo, S. Wang, and A. El Saddik. 2018. “Engineering vehicles detection based on modified faster R-CNN for power grid surveillance.” Sensors 18 (7): 2258. https://doi.org/10.3390/s18072258.
Zhou, J., C.-M. Vong, Q. Liu, and Z. Wang. 2019. “Scale adaptive image cropping for UAV object detection.” Neurocomputing 366 (Nov): 305–313. https://doi.org/10.1016/j.neucom.2019.07.073.
Zhou, Z., J. Irizarry, and Y. Lu. 2018. “A multidimensional framework for unmanned aerial system applications in construction project management.” J. Manage. Eng. 34 (3): 04018004. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000597.
Zhu, Z., X. Ren, and Z. Chen. 2017. “Integrated detection and tracking of workforce and equipment from construction job site videos.” Autom. Constr. 81 (Sep): 161–171. https://doi.org/10.1016/j.autcon.2017.05.005.
Information & Authors
Information
Published In
Practice Periodical on Structural Design and Construction
Volume 26 • Issue 4 • November 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Nov 30, 2020
Accepted: Mar 26, 2021
Published online: Jul 27, 2021
Published in print: Nov 1, 2021
Discussion open until: Dec 27, 2021
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Ivan Mutis, Pavlo Antonenko, Unmanned aerial vehicles as educational technology systems in construction engineering education, Journal of Information Technology in Construction, 10.36680/j.itcon.2022.014, 27, (273-289), (2022).