Multi-Task Deep Learning-Based Human Intention Prediction for Human-Robot Collaborative Assembly
Publication: Computing in Civil Engineering 2023
ABSTRACT
Construction robots have great potential to serve as assistants to relieve construction workers from repetitive and physically demanding tasks. It is essential for robots to understand and predict human intention in order to adapt their motion to ensure smooth human-robot collaboration. This study proposes a long short-term memory model-based multi-task learning framework to simultaneously predict multi-level human intention in assembly tasks, including high-level actions and objects, and low-level body movements, from observed body movements and associated assembly components extracted from videos. The proposed models were trained and tested using 54 videos collected with nine participants performing six assembly tasks, achieving an accuracy of 82% and 98% in action and object prediction, respectively, and an average displacement error of 8.71 pixels in pose prediction. The incorporation of work context significantly improves the accuracy of object prediction by 11.36%, with the performance of other two tasks increasing slightly.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Cai, J., Li, X., Liang, X., Wei, W., and Li, S. (2022). “Construction Worker Ergonomic Assessment via LSTM-Based Multi-Task Learning Framework.” 2022 Construction Research Congress, 215–224.
Cai, J., Yang, L., Zhang, Y., Li, S., and Cai, H. (2021). “Multitask Learning Method for Detecting the Visual Focus of Attention of Construction Workers.” Journal of Construction Engineering and Management, American Society of Civil Engineers, 147(7), 4021063.
Cardno, C. A. (2018). “Robotic Rebar-Tying System Uses Artificial Intelligence.” Civil Engineering Magazine Archive, 88(1), 38–39.
Kim, S., Peavy, M., Huang, P. C., and Kim, K. (2021). “Development of BIM-integrated construction robot task planning and simulation system.” Automation in Construction, 127.
Krämer, N. C., Von Der Pütten, A., and Eimler, S. (2012). “Human-agent and human-robot interaction theory: Similarities to and differences from human-human interaction.” Studies in Computational Intelligence, 396, 215–240.
Liu, H., and Wang, L. (2017). “Human motion prediction for human-robot collaboration.” Journal of Manufacturing Systems, 44, 287–294.
Liu, Z., Liu, Q., Xu, W., Liu, Z., Zhou, Z., and Chen, J. (2019). “Deep learning-based human motion prediction considering context awareness for human-robot collaboration in manufacturing.” Procedia CIRP, 83, 272–278.
Luo, R. C., and Mai, L. (2019). “Human Intention Inference and On-Line Human Hand Motion Prediction for Human-Robot Collaboration.” IEEE International Conference on Intelligent Robots and Systems, 5958–5964.
Madsen, A. J. (2019). “The SAM100: Analyzing Labor Productivity.” <https://digitalcommons.calpoly.edu/cmsp/243/>(Jul. 5, 2022).
Mao, W., Liu, M., Salzmann, M., and Li, H. (2019). “Learning trajectory dependencies for human motion prediction.” Proceedings of the IEEE International Conference on Computer Vision, 9488–9496.
Wang, W., Li, R., Chen, Y., Sun, Y., and Jia, Y. (2022). “Predicting Human Intentions in Human-Robot Hand-Over Tasks Through Multimodal Learning.” IEEE Transactions on Automation Science and Engineering, 19(3), 2339–2353.
Wibranek, B., and Tessmann, O. (2019). “Digital rubble compression-only structures with irregular rock and 3D printed connectors.” IASS Symposium 2019 - 60th Anniversary Symposium of the International Association for Shell and Spatial Structures; Structural Membranes 2019 - 9th International Conference on Textile Composites and Inflatable Structures, FORM and FORCE, 2488–2495.
Xiu, Y., Li, J., Wang, H., Fang, Y., and Lu, C. (2019). “Pose flow: Efficient online pose tracking.” British Machine Vision Conference 2018, BMVC 2018, BMVA Press.
Yuan, Y., and Kitani, K. (2020). “DLow: Diversifying Latent Flows for Diverse Human Motion Prediction.” Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 346–364.
Zhou, T., Wang, Y., Zhu, Q., and Du, J. (2022). “Human hand motion prediction based on feature grouping and deep learning: Pipe skid maintenance example.” Automation in Construction, 138.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 579 - 587
History
Published online: Jan 25, 2024
ASCE Technical Topics:
- Automation and robotics
- Business management
- Computer vision and image processing
- Continuum mechanics
- Displacement (mechanics)
- Dynamics (solid mechanics)
- Employment
- Engineering fundamentals
- Engineering mechanics
- Errors (statistics)
- Human and behavioral factors
- Labor
- Mathematics
- Methodology (by type)
- Model accuracy
- Models (by type)
- Motion (dynamics)
- Personnel management
- Practice and Profession
- Solid mechanics
- Statistics
- Structural mechanics
- Systems engineering
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.