Context-Aware Deep Learning Model for 3D Human Motion Prediction in Human-Robot Collaborative Construction
Publication: Computing in Civil Engineering 2023
ABSTRACT
Advances in robotics enable the implementation of collaborative robots in hazardous, repetitive, and demanding construction tasks to improve safety and productivity. Accurate and reliable human motion prediction is required to achieve smooth human-robot collaboration (HRC). However, many deep-learning-based models only consider observed movement to predict human motion while neglecting the interactions between humans and their surroundings. This study proposes a context-aware deep learning model, integrating observed movement and context information (i.e., locations of assigned tasks) into a long short-term memory network with an encoder-decoder architecture to predict a sequence of human motion in 3D. A pilot experiment was conducted, and the proposed model achieves an average displacement error of 0.15 m. The results show that incorporating task contextual information improves the accuracy of human motion prediction by 6.25%, which could augment the perception and reasoning capability of collaborative robots for improved HRC in construction.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Alahi, A., Goel, K., Ramanathan, V., Robicquet, A., Fei-Fei, L., and Savarese, S. (2016). Social LSTM: Human Trajectory Prediction in Crowded Spaces. 961–971. https://openaccess.thecvf.com/content_cvpr_2016/html/Alahi_Social_LSTM_Human_CVPR_2016_paper.html.
Bock, T. (2015). The future of construction automation: Technological disruption and the upcoming ubiquity of robotics. Automation in Construction, 59, 113–121. https://doi.org/10.1016/j.autcon.2015.07.022.
Cai, J., Zhang, Y., Yang, L., Cai, H., and Li, S. (2020). A context-augmented deep learning approach for worker trajectory prediction on unstructured and dynamic construction sites. Advanced Engineering Informatics, 46, 101173. https://doi.org/10.1016/j.aei.2020.101173.
Cui, Q., Sun, H., and Yang, F. (2020). Learning Dynamic Relationships for 3D Human Motion Prediction. 6519–6527. https://openaccess.thecvf.com/content_CVPR_2020/html/Cui_Learning_Dynamic_Relationships_for_3D_Human_Motion_Prediction_CVPR_2020_paper.html.
Dörfler, K., Sandy, T., Giftthaler, M., Gramazio, F., Kohler, M., and Buchli, J. (2016). Mobile Robotic Brickwork. In D. Reinhardt, R. Saunders, & J. Burry (Eds.), Robotic Fabrication in Architecture, Art and Design 2016 (pp. 204–217). Springer International Publishing. https://doi.org/10.1007/978-3-319-26378-6_15.
Feng, C., Xiao, Y., Willette, A., McGee, W., and Kamat, V. R. (2015). Vision guided autonomous robotic assembly and as-built scanning on unstructured construction sites. Automation in Construction, 59, 128–138. https://doi.org/10.1016/j.autcon.2015.06.002.
Kim, S., Peavy, M., Huang, P.-C., and Kim, K. (2021). Development of BIM-integrated construction robot task planning and simulation system. Automation in Construction, 127, 103720. https://doi.org/10.1016/j.autcon.2021.103720.
Liang, C.-J., Lundeen, K. M., McGee, W., Menassa, C. C., Lee, S., and Kamat, V. R. (2019). A vision-based marker-less pose estimation system for articulated construction robots. Automation in Construction, 104, 80–94. https://doi.org/10.1016/j.autcon.2019.04.004.
Liu, Z., Liu, Q., Xu, W., Liu, Z., Zhou, Z., and Chen, J. (2019). Deep Learning-based Human Motion Prediction considering Context Awareness for Human-Robot Collaboration in Manufacturing. Procedia CIRP, 83, 272–278. https://doi.org/10.1016/j.procir.2019.04.080.
Mao, W., Liu, M., Salzmann, M., and Li, H. (2019). Learning Trajectory Dependencies for Human Motion Prediction. 9489–9497. https://openaccess.thecvf.com/content_ICCV_2019/html/Mao_Learning_Trajectory_Dependencies_for_Human_Motion_Prediction_ICCV_2019_paper.html.
Martinez, J., Black, M. J., and Romero, J. (2017). On human motion prediction using recurrent neural networks. arXiv. https://doi.org/10.48550/arXiv.1705.02445.
Megalingam, R. K., Prithvi Darla, V., and Kumar Nimmala, C. S. (2020). Autonomous Wall Painting Robot. 2020 International Conference for Emerging Technology (INCET), 1–6. https://doi.org/10.1109/INCET49848.2020.9154020.
Xia, X., Zhou, T., Du, J., and Li, N. (2022). Human motion prediction for intelligent construction: A review. Automation in Construction, 142, 104497. https://doi.org/10.1016/j.autcon.2022.104497.
Xiu, Y., Li, J., Wang, H., Fang, Y., and Lu, C. (2018, February 3). Pose Flow: Efficient Online Pose Tracking. ArXiv.Org. https://arxiv.org/abs/1802.00977v2.
Yuan, Y., and Kitani, K. (2020). DLow: Diversifying Latent Flows for Diverse Human Motion Prediction. In A. Vedaldi, H. Bischof, T. Brox, & J.-M. Frahm (Eds.), Computer Vision – ECCV 2020 (pp. 346–364). Springer International Publishing. https://doi.org/10.1007/978-3-030-58545-7_20.
Zhou, T., Wang, Y., Zhu, Q., and Du, J. (2022). Human hand motion prediction based on feature grouping and deep learning: Pipe skid maintenance example. Automation in Construction, 138, 104232. https://doi.org/10.1016/j.autcon.2022.104232.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 479 - 487
History
Published online: Jan 25, 2024
ASCE Technical Topics:
- Artificial intelligence and machine learning
- Automation and robotics
- Business management
- Computer programming
- Computing in civil engineering
- Construction engineering
- Construction management
- Continuum mechanics
- Dynamics (solid mechanics)
- Engineering fundamentals
- Engineering mechanics
- Errors (statistics)
- Human and behavioral factors
- Mathematics
- Models (by type)
- Motion (dynamics)
- Neural networks
- Practice and Profession
- Public administration
- Public health and safety
- Solid mechanics
- Statistics
- Systems engineering
- Three-dimensional models
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.