Vision-Based Productivity Analysis of Cable Crane Transportation Using Augmented Reality–Based Synthetic Image
Publication: Journal of Computing in Civil Engineering
Volume 36, Issue 1
Abstract
The productivity analysis of cable crane transportation in the construction field is of great significance to improve crane equipment management and reduce operation costs. However, the traditional manual recording method of analyzing cable crane productivity is time-consuming and tedious. The existing vision-based method requires significant amounts of time to collect extensive images at construction sites and does not achieve high-precision detection in complex scenes. Thus, an automated vision-based method for productivity analysis of cable crane transportation is proposed using a new synthetic image approach based on an augmented reality (AR) technique. The unmanned aerial vehicle-based three-dimensional (3D) reconstruction of a crane bucket model is superimposed on a realistic scene using AR to synthesize the images for vision-based model training without manual image acquisition at a construction site. The feature pyramid network and attention module are integrated into Faster region-based convolutional neural network (Faster R-CNN) to enhance the capability of feature extraction for the high-precision detection of a crane bucket and its ID number, which provides the logical basis for calculating productivity. The proposed vision-based productivity analysis method is evaluated on large-scale hydraulic engineering. The results demonstrate that the mean average precision (mAP) of detection performance is 98.01% using the model trained by AR-based synthetic images, which confirms the proposed AR-based synthetic image method could provide a new image generation mode for the construction industry. Additionally, the bias of productivity between the proposed method and ground truth is 0.03%, which confirms the effectiveness and accuracy of the proposed method.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request. The following data and model can be provided from the corresponding author upon reasonable request: detection result images, synthetic images, and trained model weight files.
Acknowledgments
This research was funded by the National Natural Science Foundation of China (51839007, 51879186, and 51909187).
References
Akhavian, R., and A. H. Behzadan. 2015. “Construction equipment activity recognition for simulation input modeling using mobile sensors and machine learning classifiers.” Adv. Eng. Inf. 29 (4): 867–877. https://doi.org/10.1016/j.aei.2015.03.001.
Azuma, R. T. 1997. “A survey of augmented reality.” Presence: Teleoperators Virtual Environ. 6 (4): 355–385. https://doi.org/10.1162/pres.1997.6.4.355.
Bang, S., F. Baek, S. Park, W. Kim, and H. Kim. 2020. “Image augmentation to improve construction resource detection using generative adversarial networks, cut-and-paste, and image transformation techniques.” Autom. Constr. 115 (Jul): 103198. https://doi.org/10.1016/j.autcon.2020.103198.
Bang, S., Y. Hong, and H. Kim. 2021. “Proactive proximity monitoring with instance segmentation and unmanned aerial vehicle-acquired video-frame prediction.” Comput.-Aided Civ. Infrastruct. Eng. 36 (6): 800–816. https://doi.org/10.1111/mice.12672.
Behzadan, A. H., S. Dong, and V. R. Kamat. 2015. “Augmented reality visualization: A review of civil infrastructure system applications.” Adv. Eng. Inf. 29 (2): 252–267. https://doi.org/10.1016/j.aei.2015.03.005.
Bianchi, E., A. L. Abbott, P. Tokekar, and M. Hebdon. 2021. “COCO-bridge: Structural detail data set for bridge inspections.” J. Comput. Civ. Eng. 35 (3): 04021003. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000949.
Chen, C., Z. Zhu, and A. Hammad. 2020. “Automated excavators activity recognition and productivity analysis from construction site surveillance videos.” Autom. Constr. 110 (Feb): 103045. https://doi.org/10.1016/j.autcon.2019.103045.
Dai, J., Y. Li, K. He, and J. Sun. 2016. “R-FCN: Object detection via region-based fully convolutional networks.” In Proc., Advances in Neural Information Processing Systems, 379–387. Barcelona, Spain: Neural Information Processing Systems Foundation.
Davila Delgado, J. M., L. Oyedele, P. Demian, and T. Beach. 2020. “A research agenda for augmented and virtual reality in architecture, engineering and construction.” Adv. Eng. Inf. 45 (Aug): 101122. https://doi.org/10.1016/j.aei.2020.101122.
Deng, H., H. Hong, D. Luo, Y. Deng, and C. Su. 2020. “Automatic indoor construction process monitoring for tiles based on bim and computer vision.” J. Constr. Eng. Manage. 146 (1): 04019095. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001744.
Everingham, M., L. Van Gool, C. K. Williams, J. Winn, and A. Zisserman. 2010. “The PASCALVisual object classes (VOC) challenge.” Int. J. Comput. Vision 88 (2): 303–338. https://doi.org/10.1007/s11263-009-0275-4.
Fang, W., L. Ding, H. Luo, and P. E. D. Love. 2018. “Falls from heights: A computer vision-based approach for safety harness detection.” Autom. Constr. 91 (Jul): 53–61. https://doi.org/10.1016/j.autcon.2018.02.018.
Furukawa, Y., B. Curless, S. M. Seitz, and R. Szeliski. 2010. “Towards internet-scale multi-view stereo.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition, 1434–1441. New York: IEEE.
Gao, X. W., S. Q. Li, B. Y. Jin, M. Hu, and W. Ding. 2021. “Intelligent crack damage detection system in shield tunnel using combination of retinanet and optimal adaptive selection.” J. Intell. Fuzzy Syst. 40 (3): 4453–4469. https://doi.org/10.3233/JIFS-201296.
Gong, J., and C. H. Caldas. 2010. “Computer vision-based video interpretation model for automated productivity analysis of construction operations.” J. Comput. Civ. Eng. 24 (3): 252–263. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000027.
Guan, T., D. Zhong, B. Ren, and P. Cheng. 2015. “Construction schedule optimization for high arch dams based on real-time interactive simulation.” J. Ind. Manage. Optim. 11 (4): 1321–1342. https://doi.org/10.3934/jimo.2015.11.1321.
He, K., G. Gkioxari, P. Dollar, and R. Girshick. 2017. “Mask R-CNN.” In Proc., IEEE Int. Conf. on Computer Vision, 2980–2988. New York: IEEE.
HIKVISION. 2021. Open Hikvision. Accessed January 1, 2021. https://open.hikvision.com/download/5cda567cf47ae80dd41a54b3?type=10.
Jiang, S., and J. Zhang. 2020. “Real-time crack assessment using deep neural networks with wall-climbing unmanned aerial system.” Comput.-Aided Civ. Infrastruct. Eng. 35 (6): 549–564. https://doi.org/10.1111/mice.12519.
Kim, D., S. Lee, and V. R. Kamat. 2020. “Proximity prediction of mobile objects to prevent contact-driven accidents in co-robotic construction.” J. Comput. Civ. Eng. 34 (4): 04020022. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000899.
Kim, H., C. R. Ahn, D. Engelhaupt, and S. H. Lee. 2018a. “Application of dynamic time warping to the recognition of mixed equipment activities in cycle time measurement.” Autom. Constr. 87 (Mar): 225–234. https://doi.org/10.1016/j.autcon.2017.12.014.
Kim, H., S. Bang, H. Jeong, Y. Ham, and H. Kim. 2018b. “Analyzing context and productivity of tunnel earthmoving processes using imaging and simulation.” Autom. Constr. 92 (Aug): 188–198. https://doi.org/10.1016/j.autcon.2018.04.002.
Kim, H., and H. Kim. 2018. “3D reconstruction of a concrete mixer truck for training object detectors.” Autom. Constr. 88 (Apr): 23–30. https://doi.org/10.1016/j.autcon.2017.12.034.
Kim, H., H. Kim, Y. W. Hong, and H. Byun. 2018c. “Detecting construction equipment using a region-based fully convolutional network and transfer learning.” J. Comput. Civ. Eng. 32 (2): 04017082. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000731.
Kim, J., and S. Chi. 2020. “Multi-camera vision-based productivity monitoring of earthmoving operations.” Autom. Constr. 112 (Apr): 103121. https://doi.org/10.1016/j.autcon.2020.103121.
Kolar, Z., H. Chen, and X. Luo. 2018. “Transfer learning and deep convolutional neural networks for safety guardrail detection in 2D images.” Autom. Constr. 89 (May): 58–70. https://doi.org/10.1016/j.autcon.2018.01.003.
Krizhevsky, A., I. Sutskever, and G. E. Hinton. 2012. “ImageNet classification with deep convolutional neural networks.” Accessed January 9, 2021. https://papers.nips.cc/paper/4824-imagenet-classification-with-deep -convolutional-neural-networks.pdf.
Kumar, S. S., M. Wang, D. M. Abraham, M. R. Jahanshahi, T. Iseley, and J. C. P. Cheng. 2020. “Deep learning–based automated detection of sewer defects in CCTV videos.” J. Comput. Civ. Eng. 34 (1): 04019047. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000866.
Lecun, Y., Y. Bengio, and G. Hinton. 2015. “Deep learning.” Nature 521 (7553): 436–444. https://doi.org/10.1038/nature14539.
Li, D., Q. Xie, X. Gong, Z. Yu, J. Xu, Y. Sun, and J. Wang. 2021. “Automatic defect detection of metro tunnel surfaces using a vision-based inspection system.” Adv. Eng. Inf. 47 (Jan): 101206. https://doi.org/10.1016/j.aei.2020.101206.
Lin, J. J., A. Ibrahim, S. Sarwade, and M. Golparvar-Fard. 2021. “Bridge inspection with aerial robots: Automating the entire pipeline of visual data capture, 3D mapping, defect detection, analysis, and reporting.” J. Comput. Civ. Eng. 35 (2): 04020064. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000954.
Lin, T. Y., P. Dollár, R. Girshick, K. He, B. Hariharan, and S. Belongie. 2017a. “Feature pyramid networks for object detection.” In Proc., 30th IEEE Conf. on Computer Vision and Pattern Recognition, CVPR 2017. New York: IEEE.
Lin, T. Y., P. Goyal, R. Girshick, K. He, and P. Dollar. 2017b. “Focal loss for dense object detection.” In Proc., IEEE Int. Conf. on Computer Vision. New York: IEEE.
Lin, T.-Y., M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. 2015. “Microsoft COCO: Common objects in context.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Liu, L., W. Ouyang, X. Wang, P. Fieguth, J. Chen, X. Liu, and M. Pietikäinen. 2020. “Deep learning for generic object detection: A survey.” Preprint, submitted September 6, 2018. http://arxiv.org/abs/1809.02165v4.
Liu, W., D. Anguelov, D. Erhan, C. Szegedy, S. Reed, C. Y. Fu, and A. C. Berg. 2016. “SSD: Single shot multibox detector.” In Vol. 9905 of Proc., Computer Vision—European Conf. on Computer Vision: Lecture Notes in Computer Science, edited by B. Leibe, J. Matas, N. Sebe, and K. Welling, 21–37. Cham, Switzerland: Springer.
Lu, T., A. Huyen, L. Nguyen, J. Osborne, S. Eldin, and K. Yun. 2019. “Optimized training of deep neural network for image analysis using synthetic objects and augmented reality.” In Proc., Pattern Recognition and Tracking XXX. Bellingham, WA: International Society for Optics and Photonics.
Luo, X., H. Li, H. Wang, Z. Wu, F. Dai, and D. Cao. 2019. “Vision-based detection and visualization of dynamic workspaces.” Autom. Constr. 104 (Aug): 1–13. https://doi.org/10.1016/j.autcon.2019.04.001.
Montaser, A., and O. Moselhi. 2012. “RFID+ for tracking earthmoving operations.” In Proc., Construction Research Congress 2012: Construction Challenges in a Flat World. Reston, VA: ASCE. https://doi.org/10.1061/9780784412329.102.
Pradhananga, N., and J. Teizer. 2013. “Automatic spatio-temporal analysis of construction site equipment operations using GPS data.” Autom. Constr. 29 (Jan): 107–122. https://doi.org/10.1016/j.autcon.2012.09.004.
Rashid, K. M., and J. Louis. 2019. “Times-series data augmentation and deep learning for construction equipment activity recognition.” Adv. Eng. Inf. 42 (Oct): 100944. https://doi.org/10.1016/j.aei.2019.100944.
Rashidi, A., H. R. Nejad, and M. Maghiar. 2014. “Productivity estimation of bulldozers using generalized linear mixed models.” KSCE J. Civ. Eng. 18 (6): 1580–1589. https://doi.org/10.1007/s12205-014-0354-0.
Redmon, J., S. Divvala, R. Girshick, and A. Farhadi. 2016. “You only look once: Unified, real-time object detection.” Accessed November 27, 2020. http://pjreddie.com/yolo/.
Redmon, J., and A. Farhadi. 2018. “YOLOv3: An incremental improvement.” Preprint, submitted April 8, 2018. http://arxiv.org/abs/1804.02767.
Ren, S., K. He, R. Girshick, and J. Sun. 2015. “Faster R-CNN: Towards real-time object detection with region proposal networks.” In Proc., Advances in Neural Information Processing Systems, 91–99. Montréal: Neural Information Processing Systems Foundation.
Rezazadeh Azar, E., S. Dickinson, and B. McCabe. 2013. “Server-customer interaction tracker: Computer vision-based system to estimate dirt-loading cycles.” J. Constr. Eng. Manage. 139 (7): 785–794. https://doi.org/10.1061/(ASCE)CO.1943-7862.0000652.
Sabillon, C., A. Rashidi, B. Samanta, M. A. Davenport, and D. V. Anderson. 2020. “Audio-based bayesian model for productivity estimation of cyclic construction activities.” J. Comput. Civ. Eng. 34 (1): 04019048. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000863.
Sherafat, B., C. R. Ahn, R. Akhavian, A. H. Behzadan, M. Golparvar-Fard, H. Kim, Y.-C. Lee, A. Rashidi, and E. R. Azar. 2020. “Automated methods for activity recognition of construction workers and equipment: State-of-the-art review.” J. Constr. Eng. Manage. 146 (6): 03120002. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001843.
Shorten, C., and T. M. Khoshgoftaar. 2019. “A survey on image data augmentation for deep learning.” J. Big Data 60 (6): 1–48. https://doi.org/10.1186/s40537-019-0197-0.
Soltani, M. M., Z. Zhu, and A. Hammad. 2016. “Automated annotation for visual recognition of construction resources using synthetic images.” Autom. Constr. 62 (Feb): 14–23. https://doi.org/10.1016/j.autcon.2015.10.002.
Torres Calderon, W., D. Roberts, and M. Golparvar-Fard. 2021. “Synthesizing pose sequences from 3D assets for vision-based activity analysis.” J. Comput. Civ. Eng. 35 (1): 04020052. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000937.
Tuttas, S., A. Braun, A. Borrmann, and U. Stilla. 2016. “Evaluation of acquisition strategies for image-based construction site monitoring.” ISPRS Int. Arch. Photogramm. Remote Sens. Spatial Inf. Sci. 41 (Jun): 733–740. https://doi.org/10.5194/isprs-archives-XLI-B5-733-2016.
Tzutalin. 2015. “LabelImg.” Accessed January 1, 2021. https://github.com/tzutalin/labelImg.
Viola, P., and M. Jones. 2001. “Rapid object detection using a boosted cascade of simple features.” In Proc., IEEE Computer Society Conf. on Computer Vision and Pattern Recognition. New York: IEEE.
Wang, Z., H. Li, and X. Zhang. 2019. “Construction waste recycling robot for nails and screws: Computer vision technology and neural network approach.” Autom. Constr. 97 (Jan): 220–228. https://doi.org/10.1016/j.autcon.2018.11.009.
Wang, Z., Q. Zhang, B. Yang, T. Wu, K. Lei, B. Zhang, and T. Fang. 2021. “Vision-based framework for automatic progress monitoring of precast walls by using surveillance videos during the construction phase.” J. Comput. Civ. Eng. 35 (1): 04020056. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000933.
Woo, S., J. Park, J.-Y. Lee, and I. S. Kweon. 2018. “CBAM: Convolutional block attention module.” In Proc., European Conf. on Computer Vision (ECCV). Cham, Switzerland: Springer. https://doi.org/10.1007/978-3-030-01234-2_1.
Wu, C. 2011. “VisualSFM: A visual structure from motion system.” Accessed January 1, 2021. http://ccwu.me/vsfm/.
Wu, C. 2013. “Towards linear-time incremental structure from motion.” In Proc., 2013 Int. Conf. on 3D Vision, 3DV 2013. Piscataway, NJ: IEEE.https://doi.org/0.1109/3DV.2013.25.
Wu, H., Y. Yin, S. Wang, W. Shi, K. C. Clarke, and Z. Miao. 2017. “Optimizing GPS-guidance transit route for cable crane collision avoidance using artificial immune algorithm.” GPS Solutions 21 (2): 823–834. https://doi.org/10.1007/s10291-016-0573-6.
Xiao, B., and S.-C. Kang. 2021a. “Development of an image data set of construction machines for deep learning object detection.” J. Comput. Civ. Eng. 35 (2): 05020005. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000945.
Xiao, B., and S.-C. Kang. 2021b. “Vision-based method integrating deep learning detection for tracking multiple construction machines.” J. Comput. Civ. Eng. 35 (2): 04020071. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000957.
Xuehui, A., Z. Li, L. Zuguang, W. Chengzhi, L. Pengfei, and L. Zhiwei. 2021. “Dataset and benchmark for detecting moving objects in construction sites.” Autom. Constr. 122 (2): 103482. https://doi.org/10.1016/j.autcon.2020.103482.
Yin, X., Y. Chen, A. Bouferguene, H. Zaman, M. Al-Hussein, and L. Kurach. 2020. “A deep learning-based framework for an automated defect detection system for sewer pipes.” Autom. Constr. 109 (Jan): 102967. https://doi.org/10.1016/j.autcon.2019.102967.
Yu, X., and D. Chen. 2018. “Innovative method for the construction of cable-stayed bridges by cable crane.” Struct. Eng. Int. 28 (4): 498–505. https://doi.org/10.1080/10168664.2018.1459223.
Yu, Z., Y. Shen, and C. Shen. 2021. “A real-time detection approach for bridge cracks based on YOLOv4-FPM.” Autom. Constr. 122 (Feb): 103514. https://doi.org/10.1016/j.autcon.2020.103514.
Zeng, T., J. Wang, B. Cui, X. Wang, D. Wang, and Y. Zhang. 2021. “The equipment detection and localization of large-scale construction jobsite by far-field construction surveillance video based on improving YOLOv3 and grey wolf optimizer improving extreme learning machine.” Constr. Build. Mater. 291 (12): 123268. https://doi.org/10.1016/j.conbuildmat.2021.123268.
Zhang, A., K. C. P. Wang, B. Li, E. Yang, X. Dai, Y. Peng, Y. Fei, Y. Liu, J. Q. Li, and C. Chen. 2017. “Automated pixel-level pavement crack detection on 3D asphalt surfaces using a deep-learning network.” Comput.-Aided Civ. Infrastruct. Eng. 32 (10): 805–819. https://doi.org/10.1111/mice.12297.
Zhang, C., C. C. Chang, and M. Jamshidi. 2020. “Concrete bridge surface damage detection using a single-stage detector.” Comput.-Aided Civ. Infrastruct. Eng. 35 (4): 389–409. https://doi.org/10.1111/mice.12500.
Zhang, X., H. Gao, C. Xue, J. Zhao, and Y. Liu. 2018. “Real-time vehicle detection and tracking using improved histogram of gradient features and Kalman filters.” Int. J. Adv. Rob. Syst. 15 (1): 1–9. https://doi.org/10.1177/1729881417749949.
Zhang, Y., P. Yue, G. Zhang, T. Guan, M. Lv, and D. Zhong. 2019. “Augmented reality mapping of rock mass discontinuities and rockfall susceptibility based on unmanned aerial vehicle photogrammetry.” Remote Sens. 11 (11): 1311–1345. https://doi.org/10.3390/rs11111311.
Zheng, Z., Z. Zhang, and W. Pan. 2020. “Virtual prototyping- and transfer learning-enabled module detection for modular integrated construction.” Autom. Constr. 120 (Dec): 103387. https://doi.org/10.1016/j.autcon.2020.103387.
Zhong, B., H. Wu, L. Ding, P. E. D. Love, H. Li, H. Luo, and L. Jiao. 2019. “Mapping computer vision research in construction: Developments, knowledge gaps and implications for research.” Autom. Constr. 107 (Nov): 102919. https://doi.org/10.1016/j.autcon.2019.102919.
Zhong, D., J. Li, H. Zhu, and L. Song. 2004. “Geographic information system-based visual simulation methodology and its application in concrete dam construction processes.” J. Constr. Eng. Manage. 130 (5): 742–750. https://doi.org/10.1061/(ASCE)0733-9364(2004)130:5(742).
Zhong, D., X. Li, B. Cui, B. Wu, and Y. Liu. 2018. “Technology and application of real-time compaction quality monitoring for earth-rockfill dam construction in deep narrow valley.” Autom. Constr. 90 (Jun): 23–38. https://doi.org/10.1016/j.autcon.2018.02.024.
Zhou, Y., H. Luo, and Y. Yang. 2017. “Implementation of augmented reality for segment displacement inspection during tunneling construction.” Autom. Constr. 82 (Oct): 112–121. https://doi.org/10.1016/j.autcon.2017.02.007.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 36 • Issue 1 • January 2022
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Apr 1, 2021
Accepted: Aug 3, 2021
Published online: Sep 28, 2021
Published in print: Jan 1, 2022
Discussion open until: Feb 28, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Hao Wang, Qigui Yang, Quan Liu, Chunju Zhao, Wei Zhou, Automatic Planning for Cable Crane Operations in Concrete Dam Construction, Journal of Construction Engineering and Management, 10.1061/JCEMD4.COENG-15107, 150, 12, (2024).
- I.V. Zhabokrytskyi, ncreasing the Efficiency of Creating Augmented Reality Scenes Using Neural Networks, Èlektronnoe modelirovanie, 10.15407/emodel.44.06.069, 44, 6, (69-85), (2023).
- Kaiyang Wang, Fangyu Guo, Ruijia Zhou, Liang Qian, Implementation of augmented reality in BIM-enabled construction projects: a bibliometric literature review and a case study from China, Construction Innovation, 10.1108/CI-08-2022-0196, (2023).
- Insoo Jeong, Jeongbin Hwang, Junghoon Kim, Seokho Chi, Bon-Gang Hwang, Jinwoo Kim, Vision-Based Productivity Monitoring of Tower Crane Operations during Curtain Wall Installation Using a Database-Free Approach, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5105, 37, 4, (2023).
- Titi Sari Nurul Rachmawati, Sunkuk Kim, Unmanned Aerial Vehicles (UAV) Integration with Digital Technologies toward Construction 4.0: A Systematic Literature Review, Sustainability, 10.3390/su14095708, 14, 9, (5708), (2022).
- Yee Sye Lee, Ali Rashidi, Amin Talei, Mehrdad Arashpour, Farzad Pour Rahimian, Integration of deep learning and extended reality technologies in construction engineering and management: a mixed review method, Construction Innovation, 10.1108/CI-04-2022-0075, 22, 3, (671-701), (2022).