An Application of Cycle GAN for Creating Generated Real Training Images with 3D Excavator Pose Labels from a Synthetic Model
Publication: Construction Research Congress 2024
ABSTRACT
3D excavator poses providing the motion information of the boom, arm, and bucket in 3D space support monitoring excavator activities for safety and productivity management in earthwork. Thus, previous studies have attempted to estimate 3D excavator poses using deep learning relying on the large data with high-quality annotations, which requires time-consuming and manual processes. To address this challenge, this study proposes cycle GAN to automatically create large generated real training images with 3D pose labels from synthetic images. The proposed model is trained on 800 pairs of synthetic and real images and evaluated through pre-trained ResNet50-based 3D pose estimations. The results reveal that 3D pose model trained on generated data, reaching 0.50 m key-point loss and 8.53-degree angle loss for testing on generated images, and 9.33-degree angle loss for testing on real images, yielded better results than model trained on synthetic data (i.e., 0.64 m, 15.18-degree, and 15.39-degree, respectively). This demonstrates the effectiveness of the proposed method for generating training images from synthetic images for 3D pose estimation. This 3D pose estimated from generated images enables construction managers to monitor excavator safety and productivity in the construction sites.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Assadzadeh, A., M. Arashpour, H. Li, R. Hosseini, F. Elghaish, and S. Baduge. (2023). “Excavator 3D pose estimation using deep learning and hybrid datasets”. Advanced Engineering Informatics 55: 101875.
Bernold, L. E. (2007). “Quantitative assessment of backhoe operator skill”. Journal of construction engineering and management 133(11): 889–899.
Bobadilla, L., A. Mostafavi, T. Carmenate, and S. Bista. (2014). “Predictive assessment and proactive monitoring of struck-by safety hazards in construction sites: An information space approach”. Computing in civil and building engineering (2014): 989–996.
Chen, C., Z. Zhu, and A. Hammad. (2020). “Automated excavators activity recognition and productivity analysis from construction site surveillance videos”. Automation in construction 110: 103045.
Feng, C., K. M. Lundeen, S. Dong, and V. R. Kamat. (2015). “Vision-Based Machine Pose Estimation for Excavation Monitoring and Guidance”. Conference on Autonomous and Robotic Construction of Infrastructure.
Feng, H., C.-B. Yin, W.-W. Weng, W. Ma, J.-J. Zhou, W.-H. Jia, and Z.-L. Zhang. (2018). “Robotic excavator trajectory control using an improved GA based PID controller”. Mechanical Systems and Signal Processing 105: 153–168.
He, K., X. Zhang, S. Ren, and J. Sun. (2016). “Deep residual learning for image recognition”. Proceedings of the IEEE conference on computer vision and pattern recognition.
Li, Y., H. Wang, and X. Dong. (2020). “The denoising of desert seismic data based on cycle-GAN with unpaired data training”. IEEE Geoscience and Remote Sensing Letters 18(11): 2016–2020.
Liang, C.-J., K. M. Lundeen, W. McGee, C. C. Menassa, S. Lee, and V. R. Kamat. (2019). “Fast dataset collection approach for articulated equipment pose estimation”. Computing in Civil Engineering 2019: Data, Sensing, and Analytics, American Society of Civil Engineers Reston, VA: 146–152.
Liang, C.-J., K. M. Lundeen, W. McGee, C. C. Menassa, S. Lee, and V. R. Kamat. (2019). “A vision-based marker-less pose estimation system for articulated construction robots”. Automation in Construction 104: 80–94.
Mahmood, B., S. Han, and J. Seo. (2022). “Implementation experiments on convolutional neural network training using synthetic images for 3D pose estimation of an excavator on real images”. Automation in Construction 133: 103996.
Naghshbandi, S. N., L. Varga, and Y. Hu. (2021). “Technologies for safe and resilient earthmoving operations: A systematic literature review”. Automation in Construction 125: 103632.
Pham, H. T., M. Rafieizonooz, S. Han, and D.-E. Lee. (2021). “Current status and future directions of deep learning applications for safety management in construction”. Sustainability 13(24): 13579.
Radwan, I., A. Dhall, and R. Goecke. (2013). “Monocular image 3D human pose estimation under self-occlusion”. Proceedings of the IEEE International Conference on Computer Vision.
Richardson, A., J. Strom, and E. Olson. (2013). “AprilCal: Assisted and repeatable camera calibration”. 2013 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE.
Rogez, G., P. Weinzaepfel, and C. Schmid. (2017). “Lcr-net: Localization-classification-regression for human pose”. Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Soltani, M. M., Z. Zhu, and A. Hammad. (2018). “Framework for location data fusion and pose estimation of excavators using stereo vision”. Journal of Computing in Civil Engineering 32(6): 04018045.
Tang, J., H. Luo, W. Chen, P. K.-Y. Wong, and J. C. Cheng. (2022). “IMU-based full-body pose estimation for construction machines using kinematics modeling”. Automation in Construction 138: 104217.
Vahdatikhaki, F., A. Hammad, and H. Siddiqui. (2015). “Optimization-based excavator pose estimation using real-time location systems”. Automation in Construction 56: 76–92.
Wen, L., D. Kim, M. Liu, and S. Lee. (2022). “3D Excavator Pose Estimation Using Projection-Based Pose Optimization for Contact-Driven Hazard Monitoring”. Journal of Computing in Civil Engineering 37(1): 04022048.
Wen, L., D. Kim, M. Liu, and S. Lee. (2023). “3D Excavator Pose Estimation Using Projection-Based Pose Optimization for Contact-Driven Hazard Monitoring”. Journal of Computing in Civil Engineering 37(1): 04022048.
You, C., G. Li, Y. Zhang, X. Zhang, H. Shan, M. Li, S. Ju, Z. Zhao, Z. Zhang, and W. Cong. (2019). “CT super-resolution GAN constrained by the identical, residual, and cycle learning ensemble (GAN-CIRCLE)”. IEEE transactions on medical imaging 39(1): 188–203.
Yuan, C., S. Li, and H. Cai. (2017). “Vision-based excavator detection and tracking using hybrid kinematic shapes and key nodes”. Journal of Computing in Civil Engineering 31(1): 04016038.
Information & Authors
Information
Published In
History
Published online: Mar 18, 2024
ASCE Technical Topics:
- Automation and robotics
- Construction engineering
- Construction management
- Construction methods
- Construction sites
- Continuum mechanics
- Dynamics (solid mechanics)
- Education
- Engineering fundamentals
- Engineering mechanics
- Excavation
- Information management
- Models (by type)
- Motion (dynamics)
- Practice and Profession
- Solid mechanics
- Systems engineering
- Three-dimensional models
- Training
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.