Eye Gaze and Hand Gesture-Driven Human-Robot Interaction in Construction
Publication: Construction Research Congress 2024
ABSTRACT
Construction robots are a powerful driving force to enable intelligent processes in construction. User-friendly interfaces to support human-robot work collaboration are critical for increasing adoption of robots. Among different interfaces, eye gaze and hand gesture are effective and reliable interaction cues in the noisy construction environment. This paper proposes a novel context-aware method, which integrates eye tracking and gesture recognition for human-robot collaboration in construction. The proposed method employs a two-stream network architecture comprising a first-person view-based stream and a motion sensory data-based stream. The first-person view-based stream models the user’s gaze using an attention module to generate an attention map, which helps the stream to focus on the relevant spatiotemporal regions for context extraction. The motion sensory data-based stream is used to process the motion sensory data to extract features related to hand motions. Finally, the extracted vision context and motion features are combined to achieve the gesture recognition for conveying a message between the worker and the robot. This method was tested using a dataset gathered on construction sites. The test results show the proposed method can achieve accuracy and mean class accuracy of 96.8% and 97.7%, illustrating its effectiveness for human-robot collaboration in construction.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
ASEA Brown Boveri. 2021. “ABB Robotics advances construction industry automation to enable safer and sustainable building.” Accessed August 30, 2022. https://new.abb.com/news/detail/78359/abb-robotics-advances-construction-industry-automation-to-enable-safer-and-sustainable-building.
Barz, M., Shahzad Bhatti, O., Alam, H. M. T., Minh Ho Nguyen, D., and Sonntag, D. 2023. “Interactive Fixation-to-AOI Mapping for Mobile Eye Tracking Data based on Few-Shot Image Classification.” In Companion Proceedings of the 28th International Conference on Intelligent User Interfaces, ACM, 175–178.
BigRentz. 2020. “Crane Hand Signals to Know for a Safe Construction Site.” Accessed August 31, 2022. https://www.bigrentz.com.
Bongiovanni, A., et al. 2023. “Gestural and Touchscreen Interaction for Human-Robot Collaboration: A Comparative Study.” In International Conference on Intelligent Autonomous Systems, Cham: Springer Nature Switzerland, 122–138.
Bozomitu, R. G., Păsărică, A., Tărniceriu, D., and Rotariu, C. 2019. “Development of an eye tracking-based human-computer interface for real-time applications.” Sensors (Switzerland). 19 (16), 3630.
Carreira, J., and Zisserman, A. 2017. “Quo Vadis, action recognition? A new model and the kinetics dataset.” In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition, IEEE, 6299–6308.
Fang, B., Lv, Q., Shan, J., Sun, F., Liu, H., Guo, D., and Zhao, Y. 2019. “Dynamic gesture recognition using inertial sensors-based data gloves.” In 2019 4th IEEE International Conference on Advanced Robotics and Mechatronics, IEEE, 390–395.
FORT Robotics. 2020. “3 Ways Robots Are Making Construction Safer | FORT Robotics Industry Insights.” Accessed August 30, 2022. https://www.automate.org/industry-insights/3-ways-robots-are-making-construction-safer.
He, K., Zhang, X., Ren, S., and Sun, J. 2016. “Deep residual learning for image recognition.” In Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, IEEE, 770–778.
Hosp, B., Eivazi, S., Maurer, M., Fuhl, W., Geisler, D., and Kasneci, E. 2020. “RemoteEye: An open-source high-speed remote eye tracker.” Behavior Research Methods. 52, 1387–1401.
Hu, Z., Pan, J., Fan, T., Yang, R., and Manocha, D. 2019. “Safe Navigation with Human Instructions in Complex Scenes.” IEEE Robotics and Automation Letters. 4 (2), 753–760.
Jirak, D., Tietz, S., Ali, H., and Wermter, S. 2020. “Echo State Networks and Long Short-Term Memory for Continuous Gesture Recognition: a Comparative Study.” Cognitive Computation. 1–13.
Kim, M., Cho, J., Lee, S., and Jung, Y. 2019. “IMU sensor-based hand gesture recognition for human-machine interfaces.” Sensors (Switzerland). 19 (18), 3827.
Koller, O., Camgoz, N. C., Ney, H., and Bowden, R. 2020. “Weakly Supervised Learning with Multi-Stream CNN-LSTM-HMMs to Discover Sequential Parallelism in Sign Language Videos.” IEEE Transactions on Pattern Analysis and Machine Intelligence. 42 (9), 2306–2320.
Köpüklü, O., Gunduz, A., Kose, N., and Rigoll, G. 2019. “Real-time hand gesture detection and classification using convolutional neural networks.” In Proceedings of 14th IEEE International Conference on Automatic Face and Gesture Recognition, IEEE, 1–8.
Krishna Sharma, V., Saluja, K., Mollyn, V., and Biswas, P. 2020. “Eye gaze controlled robotic arm for persons with severe speech and motor impairment.” In Eye Tracking Research and Applications Symposium (ETRA), ACM, 1–9.
Laddi, A., and Prakash, N. R. 2019. “Eye gaze tracking based directional control interface for interactive applications.” Multimedia Tools and Applications. 78, 31215–31230.
Li, Y., Liu, M., and Rehg, J. 2021. “In the Eye of the Beholder: Gaze and Actions in First Person Video.” IEEE Transactions on Pattern Analysis and Machine Intelligence. 45 (6), 6731–6747.
Liu, H., and Wang, L. 2021. “Collision-free human-robot collaboration based on context awareness.” Robotics and Computer-Integrated Manufacturing. 67, 101997.
Lu, M., Liao, D., and Li, Z. N. 2019. “Learning spatiotemporal attention for egocentric action recognition.” In Proceedings of 2019 International Conference on Computer Vision Workshop, IEEE.
Miao, Q., Li, Y., Ouyang, W., Ma, Z., Xu, X., Shi, W., and Cao, X. 2017. “Multimodal Gesture Recognition Based on the ResC3D Network.” In Proceedings of 2017 IEEE International Conference on Computer Vision Workshops, IEEE, 3047–3055.
Multani, R. 2021. “Robotics in Construction Industry in 2022 | Use, Benefits & Types.” Linkedin. Accessed April 26, 2023. https://www.linkedin.com/pulse/robotics-construction-industry-2022-use-benefits-types-reetie-multani/.
Neacsu, A. A., Cioroiu, G., Radoi, A., and Burileanu, C. 2019. “Automatic EMG-based hand gesture recognition system using time-domain descriptors and fully-connected neural networks.” In 42nd International Conference on Telecommunications and Signal Processing, IEEE, 232–235.
Papoutsaki, A., Laskey, J., and Huang, J. 2017. “SearchGazer: Webcam eye tracking for remote studies of web search.” In Proceedings of the 2017 Conference Human Information Interaction and Retrieval, ACM, 17–26.
Petersch, B., and Dierkes, K. 2022. “Gaze-angle dependency of pupil-size measurements in head-mounted eye tracking.” Behavior Research Methods. 54 (2), 763–779.
Steil, J., Huang, M. X., and Bulling, A. 2018. “Fixation detection for head-mounted eye tracking based on visual similarity of gaze targets.” In Eye Tracking Research and Applications Symposium, ACM, 1–9.
Su, H., Ovur, S. E., Zhou, X., Qi, W., Ferrigno, G., and De Momi, E. 2020. “Depth vision guided hand gesture recognition using electromyographic signals.” Advanced Robotics. 34 (15), 985–997.
Tap Systems Inc. 2021. “Meet Tap.” Accessed February 3, 2022. https://www.tapwithus.com/.
Tobii Inc. 2021. “Tobii Pro Glasses 3.” Accessed February 3, 2022. https://www.tobiipro.com/product-listing/tobii-pro-glasses-3/.
Wang, X., Veeramani, D., and Zhu, Z. 2023a. “Wearable Sensors-Based Hand Gesture Recognition for Human-Robot Collaboration in Construction.” IEEE Sensors Journal. 23 (1), 495–505.
Wang, X., Veeramani, D., and Zhu, Z. 2023b. “Gaze-aware hand gesture recognition for intelligent construction.” Engineering Applications of Artificial Intelligence. 123 (4), 106179.
Wang, X., and Zhu, Z. 2021a. “Vision–Based Framework for Automatic Interpretation of Construction Workers’ Hand Gestures.” Automation in Construction. 130, 103872.
Wang, X., and Zhu, Z. 2021b. “Vision-Based Hand Signal Recognition in Construction: A Feasibility Study.” Automation in Construction. 125, 103625.
Wang, Z., Yan, W., and Oates, T. 2017. “Time series classification from scratch with deep neural networks: A strong baseline.” In Proceedings of the International Joint Conference on Neural Networks, IEEE, 1578–1585.
Information & Authors
Information
Published In
Construction Research Congress 2024
Pages: 991 - 1000
History
Published online: Mar 18, 2024
ASCE Technical Topics:
- Automation and robotics
- Business management
- Construction engineering
- Construction management
- Construction methods
- Continuum mechanics
- Dynamics (solid mechanics)
- Engineering fundamentals
- Engineering mechanics
- Equipment and machinery
- Human and behavioral factors
- Motion (dynamics)
- Practice and Profession
- Probe instruments
- River engineering
- Rivers and streams
- Solid mechanics
- Systems engineering
- Water and water resources
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.