A New Benchmark Model for the Automated Detection and Classification of a Wide Range of Heavy Construction Equipment
Publication: Journal of Management in Engineering
Volume 40, Issue 2
Abstract
The integration of computer vision technology into construction sites poses various challenges due to the complex environment. Prior studies on computer vision related to heavy construction equipment has primarily focused on a limited range of equipment types provided in standard databases, such as the Microsoft Common Objects in Context (MS COCO) data set. The conventional approach has limitations in capturing the diverse working conditions and dynamic environments encountered in real construction sites. To overcome the challenge, this study proposes a new benchmark model for the automated detection and classification of a wide range of heavy construction equipment (i.e., nine representative types) commonly used in construction sites by using a deep convolution neural network. This study was conducted in four steps: (1) data collection and preparation, (2) data transformation, (3) model training, and (4) model validation. The proposed you only look once (YOLO)v5l (large, YOLOv5 with a larger network) model demonstrated high reliability, achieving a mean average precision of 90.26%. This study makes a significant contribution to the domain of construction engineering and management by providing a more efficient and systematic management system to proactively prevent heavy equipment–related safety accidents with diverse working conditions and dynamic environments encountered at construction sites. Moreover, the proposed approach can be extended to integrate advanced techniques such as case-based reasoning, digital twin, and blockchain, allowing for the automated activity recognition in various occlusions, the carbon emissions monitoring and diagnostics of heavy equipment, and a robust real-time construction management system with enhanced security.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was supported by the National Research Foundation of Korea (NRF) grant funded by the Korea government (MSIT) (No. RS-2023-00217322).
References
Akinosho, T. D., L. O. Oyedele, M. Bilal, A. O. Ajayi, M. D. Delgado, O. O. Akinade, and A. A. Ahmed. 2020. “Deep learning in the construction industry: A review of present status and future innovations.” J. Build. Eng. 32 (Nov): 101827. https://doi.org/10.1016/j.jobe.2020.101827.
Austin, M., P. Delgoshaei, M. Coelho, and M. Heidarinejad. 2020. “Architecting smart city digital twins: Combined semantic model and machine learning approach.” J. Manage. Eng. 36 (4): 04020026. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000774.
Azar, E. R., and B. McCabe. 2012. “Part based model and spatial–temporal reasoning to recognize hydraulic excavators in construction images and videos.” Autom. Constr. 24 (Jul): 194–202. https://doi.org/10.1016/j.autcon.2012.03.003.
Blue, S. T., and M. Brindha. 2019. “Edge detection based boundary box construction algorithm for improving the precision of object detection in YOLOv3.” In Proc., 2019 10th Int. Conf. on Computing, Communication and Networking Technologies (ICCCNT), 1–5. New York: IEEE.
Chen, C., Z. Zhu, and A. Hammad. 2020. “Automatic analysis of idling in excavator’s operations based on excavator-truck relationships.” Autom. Rob. Constr. 37 (Mar): 1307–1313. https://doi.org/10.22260/ISARC2020/0179.
Chen, C., Z. Zhu, and A. Hammad. 2022. “Critical review and road map of automated methods for earthmoving equipment productivity monitoring.” J. Comput. Civ. Eng. 36 (3): 03122001. https://doi.org/10.1061/(ASCE)CP.1943-5487.0001017.
Chen, G., M. Liu, Y. Zhang, Z. Wang, S. M. Hsiang, and C. He. 2023. “Using images to detect, plan, analyze, and coordinate a smart contract in construction.” J. Manage. Eng. 39 (2): 04023002. https://doi.org/10.1061/JMENEA.MEENG-5121.
Cho, Y., K. Kang, B. Son, and H. Ryu. 2021. “Extraction of workers and heavy equipment and muliti-object tracking using surveillance system in construction sites.” J. Korea Inst. Build. Constr. 21 (5): 397–408. https://doi.org/10.5345/JKIBC.2021.21.5.397.
Duan, R., H. Deng, M. Tian, Y. Deng, and J. Lin. 2022. “SODA: A large-scale open site object detection dataset for deep learning in construction.” Autom. Constr. 142 (Mar): 104499. https://doi.org/10.1016/j.autcon.2022.104499.
Fang, W., L. Ding, B. Zhong, P. E. D. Love, and H. Luo. 2018. “Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach.” Adv. Eng. Inf. 37 (Mar): 139–149. https://doi.org/10.1016/j.aei.2018.05.003.
Flaticon. 2023. “Icons mineral supply chains.” Accessed November 20, 2023. https://www.flaticon.com/.
Golparvar-Fard, M., A. Heydarian, and J. C. Niebles. 2013. “Vision-based action recognition of earthmoving equipment using spatio-temporal features and support vector machine classifiers.” Adv. Eng. Inf. 27 (4): 652–663. https://doi.org/10.1016/j.aei.2013.09.001.
Guo, B. H. W., Y. Zou, Y. Fang, Y. M. Goh, and P. X. W. Zou. 2021. “Computer vision technologies for safety science and management in construction: A critical review and future research directions.” Saf. Sci. 135 (Apr): 105130. https://doi.org/10.1016/j.ssci.2020.105130.
Hou, L., C. Chen, S. Wang, Y. Wu, and X. Chen. 2022. “Multi-object detection method in construction machinery swarm operations based on the improved YOLOv4 model.” Sensors 22 (19): 7294. https://doi.org/10.3390/s22197294.
Kim, J., S. Chi, and J. Seo. 2018. “Interaction analysis for vision-based activity identification of earthmoving excavators and dump trucks.” Autom. Constr. 87 (Mar): 297–308. https://doi.org/10.1016/j.autcon.2017.12.016.
Kim, S., K. Jeong, T. Hong, J. Lee, and J. Lee. 2023. “Deep learning–based automated generation of material data with object–space relationships for scan to BIM.” J. Manage. Eng. 39 (3): 04023004. https://doi.org/10.1061/JMENEA.MEENG-5143.
KOSHA (Korea Occupational Safety and Health Agency). 2021. Major accident issue report (2021.12). Ulsan, South Korea: KOSHA.
Kropp, C., C. Koch, and M. König. 2018. “Interior construction state recognition with 4D BIM registered image sequences.” Autom. Constr. 86 (1): 11–32. https://doi.org/10.1016/j.autcon.2017.10.027.
Ku, B., K. Kim, and J. Jeong. 2022. “Real-Time ISR-YOLOv4 Based small object detection for safe shop floor in smart factories.” Electronics 11 (15): 2348. https://doi.org/10.3390/electronics11152348.
Kumar, S., H. Gupta, D. Yadav, I. A. Ansari, and O. P. Verma. 2022. “YOLOv4 algorithm for the real-time detection of fire and personal protective equipments at construction sites.” Multimedia Tools Appl. 81 (16): 22163–22183. https://doi.org/10.1007/s11042-021-11280-6.
Lee, D. J., and S. K. Kim. 2022. “Factors of selecting temporary road positions for the optimal path of earthwork equipment in road constructions.” Korean J. Constr. Eng. Manage. 23 (2): 85–94. https://doi.org/10.6106/KJCEM.2022.23.2.085.
Lee, T., M. Cho, H. Kim, T. Kim, and H. Jung. 2021. “Implementation of a model for predicting the distance between hazardous object and workers in the workplace using YOLO-v4.” In Vol. 25 of Proc., Korean Institute of Information and Communication Sciences Conf., 332–334. Seoul: Korea Institute of Information and Communication Engineering.
Lee, W., S. Kim, Y. Yu, and B. Koo. 2022. “Development of graph based deep learning methods for enhancing the semantic integrity of spaces in BIM models.” Korean J. Constr. Eng. Manage. 23 (3): 45–55. https://doi.org/10.6106/KJCEM.2022.23.3.045.
Luo, X., H. Li, D. Cao, F. Dai, J. Seo, and S. Lee. 2018. “Recognizing diverse construction activities in site images via relevance networks of construction-related objects detected by convolutional neural networks.” J. Comput. Civ. Eng. 32 (3): 04018012. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000756.
Mei, X., X. Zhou, F. Xu, and Z. Zhang. 2023. “Human intrusion detection in static hazardous areas at construction sites: Deep learning–based method.” J. Constr. Eng. Manage. 149 (1): 04022142. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002409.
MOFL (Ministry of Employment and Labor). 2021. Self-checklist for preventing major industrial accidents in the construction industry. Sejong, South Korea: MOFL.
Nath, N. D., and A. H. Behzadan. 2020. “Deep convolutional networks for construction object detection under different visual conditions.” Front. Built Environ. 6 (Aug): 97. https://doi.org/10.3389/fbuil.2020.00097.
Pal, A., and S. H. Hsieh. 2021. “Deep-learning-based visual data analytics for smart construction management.” Autom. Constr. 131 (Mar): 103892. https://doi.org/10.1016/j.autcon.2021.103892.
Park, J., J. Kim, D. Lee, K. Jeong, J. Lee, H. Kim, and T. Hong. 2022a. “Deep learning–based automation of scan-to-BIM with modeling objects from occluded point clouds.” J. Manage. Eng. 38 (4): 04022025. https://doi.org/10.1061/(ASCE)ME.1943-5479.0001055.
Park, J., S. Park, J. Kim, and J. Kim. 2022b. “A vision-based pipe support displacement measurement method using moire patterns.” Korean J. Constr. Eng. Manage. 23 (1): 37–45. https://doi.org/10.6106/KJCEM.2022.23.1.037.
Park, M. W., N. Elsafty, and Z. Zhu. 2015. “Hardhat-wearing detection for enhancing on-site safety of construction workers.” J. Constr. Eng. Manage. 141 (9): 04015024. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000974.
Reja, V. K., K. Varghese, and Q. P. Ha. 2022. “Computer vision-based construction progress monitoring.” Autom. Constr. 138 (Jun): 104245. https://doi.org/10.1016/j.autcon.2022.104245.
Roberts, D., and M. Golparvar-Fard. 2019. “End-to-end vision-based detection, tracking and activity analysis of earthmoving equipment filmed at ground level.” Autom. Constr. 105 (Sep): 102811. https://doi.org/10.1016/j.autcon.2019.04.006.
Seong, H., J. Kim, and H. Kang. 2022. “Automated measurement method for construction errors of reinforced concrete pile foundation using a drones.” Korean J. Constr. Eng. Manage. 23 (2): 45–53. https://doi.org/10.6106/KJCEM.2022.23.2.045.
Shim, S., and S. I. Choi. 2019. “Development on identification algorithm of risk situation around construction vehicle using YOLO-v3.” J. Korea Acad. Ind. Cooperation Soc. 20 (7): 622–629. https://doi.org/10.5762/KAIS.2019.20.7.622.
Su, Y., C. Mao, R. Jiang, G. Liu, and J. Wang. 2021. “Data-driven fire safety management at building construction sites: Leveraging CNN.” J. Manage. Eng. 37 (2): 04020108. https://doi.org/10.1061/(ASCE)ME.1943-5479.0000877.
Wang, M., P. Wong, H. Luo, S. Kumar, V. Delhi, and J. Cheng. 2019. “Predicting safety hazards among construction workers and equipment using computer vision and deep learning techniques.” In Vol. 36 of Proc., Int. Symp. on Automation and Robotics in Construction, 399–406. Edinburgh, UK: International Association for Automation and Robotics in Construction. https://doi.org/10.22260/ISARC2019/0054.
Information & Authors
Information
Published In
Journal of Management in Engineering
Volume 40 • Issue 2 • March 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Apr 9, 2023
Accepted: Sep 26, 2023
Published online: Dec 21, 2023
Published in print: Mar 1, 2024
Discussion open until: May 21, 2024
ASCE Technical Topics:
- Automation and robotics
- Benchmark
- Business management
- Computer models
- Computer vision and image processing
- Construction engineering
- Construction equipment
- Construction management
- Construction sites
- Engineering fundamentals
- Equipment and machinery
- Management methods
- Methodology (by type)
- Models (by type)
- Practice and Profession
- Systems engineering
- Systems management
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.