A Deictic Gesture-Based Human-Robot Interface for In Situ Task Specification in Construction
Publication: Computing in Civil Engineering 2023
ABSTRACT
Despite the potential of robotic systems for automating the construction industry, the role of human operators remains essential for the success of these systems in complex and dynamic environments. However, current human-robot interfaces are often limited to low-level interactions that require constant micromanaging of robot movements. To address this limitation, this study proposes a deictic gesture-based interface that enables high-level task specification for construction robots. To evaluate the user experience and task performance of the proposed interface, we conducted a laboratory experiment with six human subjects who interacted with the robot to make openings in drywall panels. The results show that the proposed interface significantly reduced mental demand and effort levels among the participants compared to the conventional joystick interface. Moreover, task performance using the proposed interface was comparable in accuracy and efficiency to that achieved with the joystick interface. These findings highlight the potential of the proposed deictic gesture-based interface to facilitate intuitive human-robot interaction and precise operation of construction robots, particularly in situations where as-planned building models are not readily available.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Alibali, M. W. 2005. “Gesture in spatial cognition: Expressing, communicating, and thinking about spatial information.” Spatial Cognition and Computation. Lawrence Erlbaum Associates, Inc.
Casiez, G., N. Roussel, and D. Vogel. 2012. “1 € filter: a simple speed-based low-pass filter for noisy input in interactive systems.” Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, CHI ’12, 2527–2530. New York, NY, USA: Association for Computing Machinery.
Chen, X., X. Huang, Y. Wang, and X. Gao. 2020. “Combination of Augmented Reality Based Brain- Computer Interface and Computer Vision for High-Level Control of a Robotic Arm.” IEEE Trans. Neural Syst. Rehabil. Eng., 28 (12): 3140–3147. https://doi.org/10.1109/TNSRE.2020.3038209.
Choi, A., M. K. Jawed, and J. Joo. 2022. “Preemptive Motion Planning for Human-to-Robot Indirect Placement Handovers.”.
Feng, C., N. Fredricks, and V. R. Kamat. 2013. “Human-robot integration for pose estimation and semi-autonomous navigation on unstructured construction sites.” Proceedings of the 30th International Symposium on Automation and Robotics in Construction and Mining (ISARC 2013): Building the Future in Automation and Robotics. International Association for Automation and Robotics in Construction (IAARC).
Guzzi, J., G. Abbate, A. Paolillo, and A. Giusti. 2022. “Interacting with a Conveyor Belt in Virtual Reality using Pointing Gestures.” Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’22, 1194–1195. IEEE Press.
Han, J., G. Ajaykumar, Z. Li, and C.-M. Huang. 2020. “Structuring Human-Robot Interactions via Interaction Conventions.” 2020 29th IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), 341–348. ieeexplore.ieee.org.
Kazhdan, M., M. Bolitho, and H. Hoppe. 2006. “Poisson surface reconstruction.” Proceedings of the fourth Eurographics symposium on Geometry processing. cse.iitd.ac.in.
Labbe, M., and F. Michaud. 2014. “Online global loop closure detection for large-scale multi-session graph-based SLAM.” ieeexplore.ieee.org.
Liang, C.-J., X. Wang, V. R. Kamat, and C. C. Menassa. 2021. “Human–Robot Collaboration in Construction: Classification and Research Trends.” J. Constr. Eng. Manage., 147 (10): 03121006. American Society of Civil Engineers. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002154.
Liu, Y., M. Habibnezhad, and H. Jebelli. 2021a. “Brain-computer interface for hands-free teleoperation of construction robots.” Autom. Constr., 123: 103523. Elsevier. https://doi.org/10.1016/j.autcon.2020.103523.
Liu, Y., M. Habibnezhad, and H. Jebelli. 2021b. “Brainwave-driven human-robot collaboration in construction.” Autom. Constr., 124: 103556. Elsevier B.V. https://doi.org/10.1016/j.autcon.2021.103556.
Losey, D. P., H. J. Jeon, M. Li, K. Srinivasan, A. Mandlekar, A. Garg, J. Bohg, and D. Sadigh. 2022. “Learning latent actions to control assistive robots.” Auton. Robots, 46 (1): 115–147. https://doi.org/10.1007/s10514-021-10005-w.
Neto, P., M. Simão, N. Mendes, and M. Safeea. 2019. “Gesture-based human-robot interaction for human assistance in manufacturing.” Int. J. Adv. Manuf. Technol., 101 (1): 119–135. https://doi.org/10.1007/s00170-018-2788-x.
Sprute, D., K. Tönnies, and M. König. 2019. “This Far, No Further: Introducing Virtual Borders to Mobile Robots Using a Laser Pointer.” 2019 Third IEEE International Conference on Robotic Computing (IRC), 403–408. Institute of Electrical and Electronics Engineers Inc.
Stoddard, B., M. Cravetz, T. Player, and H. Knight. 2022. “A Haptic Multimodal Interface with Abstract Controls for Semi-Autonomous Manipulation.” Proceedings of the 2022 ACM/IEEE International Conference on Human-Robot Interaction, HRI ’22, 1206–1207. IEEE Press.
Villani, V., F. Pini, F. Leali, and C. Secchi. 2018. “Survey on human–robot collaboration in industrial settings: Safety, intuitive interfaces and applications.” Mechatronics, 55: 248–266. https://doi.org/10.1016/j.mechatronics.2018.02.009.
Wang, X., C.-J. Liang, C. C. Menassa, and V. R. Kamat. 2021. “Interactive and immersive process-level digital twin for collaborative human–robot construction work.” J. Comput. Civ. Eng., 35 (6): 04021023. American Society of Civil Engineers (ASCE). https://doi.org/10.1061/(asce)cp.1943-5487.0000988.
Wang, X., D. Veeramani, and Z. Zhu. 2023. “Wearable Sensors-Based Hand Gesture Recognition for Human–Robot Collaboration in Construction.” IEEE Sens. J., 23 (1): 495–505. https://doi.org/10.1109/JSEN.2022.3222801.
Wang, X., and Z. Zhu. 2021. “Vision–based framework for automatic interpretation of construction workers’ hand gestures.” Autom. Constr., 130: 103872. Elsevier B.V. https://doi.org/10.1016/j.autcon.2021.103872.
Yoon, S., Y. Kim, C. Ahn, and M. Park. 2021. “Challenges in deictic gesture-based spatial referencing for human-robot interaction in construction.” Proceedings of the 38th International Symposium on Automation and Robotics in Construction (ISARC), 491–497. Waterloo, Canada, Waterloo: International Association for Automation and Robotics in Construction (IAARC).
Zhu, Q., J. Du, Y. Shi, and P. Wei. 2021. “Neurobehavioral assessment of force feedback simulation in industrial robotic teleoperation.” Autom. Constr., 126: 103674. https://doi.org/10.1016/j.autcon.2021.103674.
Zhu, Q., T. Zhou, P. Xia, and J. Du. 2022. “Robot Planning for Active Collision Avoidance in Modular Construction: Pipe Skids Example.” J. Constr. Eng. Manage., 148 (10): 04022114. American Society of Civil Engineers. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002374.
Zimmermann, C., T. Welschehold, C. Dornhege, W. Burgard, and T. Brox. 2018. “3D Human Pose Estimation in RGBD Images for Robotic Task Learning.” 2018 IEEE International Conference on Robotics and Automation (ICRA), 1986–1992. IEEE Press.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 445 - 452
History
Published online: Jan 25, 2024
ASCE Technical Topics:
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.