Human Intent Prediction in Human-Robot Collaboration—A Pipe Maintenance Example
Publication: Construction Research Congress 2022
ABSTRACT
Human-robot collaboration has gained popularity in various civil engineering applications. The key to a successful human-robot collaboration is the design of an intelligent robot system that is aware of human intents and can predict human motions. Despite the advances in human intent prediction in the context of human-robot collaboration, challenges still present. Most intelligent systems can only predict human motions based on the aggregated data from different human subjects. A method that is capable of capturing the changing characteristics of human motions and predict human motions in dynamic and open workplaces is needed. This paper proposes an innovative analytical method that predicts human motions using the Light Gradient Boosting Machine (LightGBM) with incremental learning. A virtual reality-based human subject experiment (n = 120) was performed to collect the gaze tracking data and the corresponding two-hands motion data in a pipe maintenance task. First, the relationship between the gaze focus and the hand motion is explored via the symbolic aggregate approximation (SAX) to identify the latency between a person’s gaze focus direction and the hand motions. Then, the continuous time-series data of gaze focus is used to predict the motions of the hand, with eye-hand delay adjustments incorporated. The proposed method can significantly improve the accuracy of human motion prediction in a complex pipe maintenance task, and thus benefit a better design of collaborative robotic systems.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Borbély, B. J., and Szolgay, P. (2017). “Real-time inverse kinematics for the upper limb: a model-based algorithm using segment orientations.” Biomedical engineering online, 16(1), 21.
Botchkarev, A. (2018). “Performance metrics (error measures) in machine learning regression, forecasting and prognostics: Properties and typology.”.
Bruce, A., and Gordon, G. “Better motion prediction for people-tracking.” Proc., Proc. of the Int. Conf. on Robotics & Automation (ICRA), Barcelona, Spain.
Butepage, J., Black, M. J., Kragic, D., and Kjellstrom, H. “Deep representation learning for human motion prediction and classification.” Proc., Proceedings of the IEEE conference on computer vision and pattern recognition, 6158–6166.
Cheng, Y., Zhao, W., Liu, C., and Tomizuka, M. “Human motion prediction using semi-adaptable neural networks.” Proc., 2019 American Control Conference (ACC), IEEE, 4884–4890.
Drury, J. L., Scholtz, J., and Yanco, H. A. “Awareness in human-robot interactions.” Proc., SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme-System Security and Assurance (Cat. No. 03CH37483), IEEE, 912–918.
Du, J., Shi, Y., Mei, C., Quarles, J., and Yan, W. “Communication by interaction: A multiplayer VR environment for building walkthroughs.” Proc., Construction Research Congress 2016, 2281–2290.
Du, J., Shi, Y., Zou, Z., and Zhao, D. (2018). “CoVR: Cloud-based multiuser virtual reality headset system for project communication of remote users.” Journal of Construction Engineering and Management, 144(2), 04017109.
Du, J., Zou, Z., Shi, Y., and Zhao, D. (2017). “Simultaneous data exchange between BIM and VR for collaborative decision making.” Computing in Civil Engineering 2017, 1–8.
Du, J., Zou, Z., Shi, Y., and Zhao, D. (2018). “Zero latency: Real-time synchronization of BIM data in virtual reality for collaborative decision-making.” Automation in Construction, 85, 51–64.
Ellis, C. A., and Parbery, S. A. (2005). “Is smarter better? A comparison of adaptive, and simple moving average trading strategies.” Research in International Business and Finance, 19(3), 399–411.
Goodrich, M. A., and Olsen, D. R. “Seven principles of efficient human robot interaction.” Proc., SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme-System Security and Assurance (Cat. No. 03CH37483), IEEE, 3942–3948.
Goodrich, M. A., and Schultz, A. C. (2008). Human-robot interaction: a survey, Now Publishers Inc.
Grosz, B. J., and Hunsberger, L. (2006). “The dynamics of intention in collaborative activity.” Cognitive Systems Research, 7(2-3), 259–272.
Haddadin, S., Albu-Schaffer, A., De Luca, A., and Hirzinger, G. “Collision detection and reaction: A contribution to safe physical human-robot interaction.” Proc., 2008 IEEE/RSJ International Conference on Intelligent Robots and Systems, IEEE, 3356–3363.
Huang, D.-A., and Kitani, K. M. “Action-reaction: Forecasting the dynamics of human interaction.” Proc., European Conference on Computer Vision, Springer, 489–504.
Ke, G., Meng, Q., Finley, T., Wang, T., Chen, W., Ma, W., Ye, Q., and Liu, T.-Y. “Lightgbm: A highly efficient gradient boosting decision tree.” Proc., Advances in neural information processing systems, 3146–3154.
Keogh, E. J., and Pazzani, M. J. “Scaling up dynamic time warping for datamining applications.” Proc., Proceedings of the sixth ACM SIGKDD international conference on Knowledge discovery and data mining, 285–289.
Kim, P., Park, J., Cho, Y. K., and Kang, J. (2019). “UAV-assisted autonomous mobile robot navigation for as-is 3D data collection and registration in cluttered environments.” Automation in Construction, 106, 102918.
Kohler, M. (1997). Using the Kalman filter to track human interactive motion: modelling and initialization of the Kalman filter for translational motion, Citeseer.
Kovar, L., Gleicher, M., and Pighin, F. (2008). “Motion graphs.” ACM SIGGRAPH 2008 classes, 1–10.
Lasota, P. A., and Shah, J. A. (2015). “Analyzing the effects of human-aware motion planning on close-proximity human–robot collaboration.” Human factors, 57(1), 21–33.
Lin, J., Keogh, E., Wei, L., and Lonardi, S. (2007). “Experiencing SAX: a novel symbolic representation of time series.” Data Mining and knowledge discovery, 15(2), 107–144.
Liu, H., and Wang, L. (2017). “Human motion prediction for human-robot collaboration.” Journal of Manufacturing Systems, 44, 287–294.
Martinez, J., Black, M. J., and Romero, J. “On human motion prediction using recurrent neural networks.” Proc., Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, 2891–2900.
Meltzoff, A. N., and Brooks, R. (2001). “Like me” as a building block for understanding other minds: Bodily acts, attention, and intention.” Intentions and intentionality: Foundations of social cognition, 171191.
Paden, B., Čáp, M., Yong, S. Z., Yershov, D., and Frazzoli, E. (2016). “A survey of motion planning and control techniques for self-driving urban vehicles.” IEEE Transactions on intelligent vehicles, 1(1), 33–55.
Pavlovic, V., Rehg, J. M., and MacCormick, J. “Learning switching linear models of human motion.” Proc., NIPS, 4.
Peng, Z., Genewein, T., and Braun, D. A. (2014). “Assessing randomness and complexity in human motion trajectories through analysis of symbolic sequences.” Frontiers in human neuroscience, 8, 168.
Powell, M. J. (1994). “A direct search optimization method that models the objective and constraint functions by linear interpolation.” Advances in optimization and numerical analysis, Springer, 51–67.
Roberts, D., Bretl, T., and Golparvar-Fard, M. (2017). “Detecting and classifying cranes using camera-equipped UAVs for monitoring crane-related safety hazards.” Computing in Civil Engineering 2017, 442–449.
Rousseeuw, P. J. (1987). “Silhouettes: a graphical aid to the interpretation and validation of cluster analysis.” Journal of computational and applied mathematics, 20, 53–65.
Rudenko, A., Palmieri, L., Herman, M., Kitani, K. M., Gavrila, D. M., and Arras, K. O. (2019). “Human motion trajectory prediction: A survey.” The International Journal of Robotics Research, 0278364920917446.
Shi, Y., Du, J., Ragan, E., Choi, K., and Ma, S. “Social influence on construction safety behaviors: a multi-user virtual reality experiment.” Proc., Construction Research Congress, 147–183.
Su, H., Yang, C., Ferrigno, G., and De Momi, E. (2019). “Improved human–robot collaborative control of redundant robot for teleoperated minimally invasive surgery.” IEEE Robotics and Automation Letters, 4(2), 1447–1453.
Thorndike, R. L. (1953). “Who belongs in the family?” Psychometrika, 18(4), 267–276.
Tibshirani, R., Walther, G., and Hastie, T. (2001). “Estimating the number of clusters in a data set via the gap statistic.” Journal of the Royal Statistical Society: Series B (Statistical Methodology), 63(2), 411–423.
Unhelkar, V. V., Lasota, P. A., Tyroller, Q., Buhai, R.-D., Marceau, L., Deml, B., and Shah, J. A. (2018). “Human-aware robotic assistant for collaborative assembly: Integrating human motion prediction with planning in time.” IEEE Robotics and Automation Letters, 3(3), 2394–2401.
Wagstaff, K., Cardie, C., Rogers, S., and Schrödl, S. “Constrained k-means clustering with background knowledge.” Proc., Icml, 577–584.
Yu, M., Lin, Y., Schmidt, D., Wang, X., and Wang, Y. (2014). “Human-robot interaction based on gaze gestures for the drone teleoperation.” Journal of Eye Movement Research, 7(4), 1–14.
Zhou, T., Zhu, Q., and Du, J. (2020). “Intuitive Robot Teleoperation for Civil Engineering Operations with Virtual Reality and Deep Learning Scene Reconstruction.” Advanced Engineering Informatics, Under Review.
Zhu, Q., Du, J., Shi, Y., and Wei, P. (2021). “Neurobehavioral assessment of force feedback simulation in industrial robotic teleoperation.” Automation in Construction, 126, 103674.
Information & Authors
Information
Published In
Construction Research Congress 2022
Pages: 581 - 590
History
Published online: Mar 7, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.