Augmented Telepresence: Enhancing Robot Arm Control with Mixed Reality for Dexterous Manipulation
Publication: Construction Research Congress 2024
ABSTRACT
Robotized agents are becoming increasingly important in the construction industry as an extension of human capabilities. In many applications, the complexity of a teleoperation task could only be completed alongside the same physical environment to ensure synchronized situational awareness. With the recent advances in mixed reality (MR) and reality capture technologies, a new interface for teleoperating robots with a physical world reference is made possible. This paper introduces an intuitive MR control interface that could mitigate the gap between humans and robots by using a HoloLens2 optical see-through head mounted display (HMD), a Franka Panda 7-DOF robot arm, ranging sensors, and Unity game engine. With real-time acquired spatial information, a high-fidelity digital twin of the physical workspace is teleported to the user in mixed reality. Simultaneously, a virtual space around the user is also defined by the HMD with their hand gesture processed as input commands. The two virtualized spaces are then registered together and merged to be organized by the Unity game engine. As a result, the user takes control of the robot’s arm through direct manipulation of the virtual/real version of their hand. The system is tested with intricate pipe fitting tasks to validate its performance.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Benotsmane, R., Dudás, L., and Kovács, G. (2020). “Trajectory optimization of industrial robot arms using a newly elaborated “whip-lashing” method.” Applied Sciences, 10(23), p. 8666.
Biess, A., Liebermann, D. G., and Flash, T. (2007). “A computational model for redundant human three-dimensional pointing movements: integration of independent spatial and temporal motor plans simplifies movement dynamics.” Journal of Neuroscience, 27(48), pp. 13045–13064.
Cai, S., Ma, Z., Skibniewski, M. J., and Bao, S. (2019). “Construction automation and robotics for high-rise buildings over the past decades: A comprehensive review.” Advanced Engineering Informatics, 42, p. 100989.
Deng, Y., Tang, Y., Yang, B., Zheng, W., Liu, S., and Liu, C. “A Review of Bilateral Teleoperation Control Strategies with Soft Environment.” Proc., 2021 6th IEEE International Conference on Advanced Robotics and Mechatronics (ICARM), IEEE, pp. 459–464.
Di Luca, M., and Mahnan, A. “Perceptual limits of visual-haptic simultaneity in virtual reality interactions.” Proc., 2019 IEEE World Haptics Conference (WHC), IEEE, pp. 67–72.
Drury, J. L., Scholtz, J., and Yanco, H. A. “Awareness in human-robot interactions.” Proc., SMC'03 Conference Proceedings. 2003 IEEE International Conference on Systems, Man and Cybernetics. Conference Theme-System Security and Assurance (Cat. No. 03CH37483), IEEE, pp. 912–918.
Emika, F. (2021). Franka Emika Robot’s Instruction Handbook, Franka Emika GmbH.
Halder, S., and Afsari, K. (2023). “Robots in inspection and monitoring of buildings and infrastructure: a systematic review.” Applied Sciences, 13(4), p. 2304.
Jankowski, J., and Grabowski, A. (2015). “Usability evaluation of vr interface for mobile robot teleoperation.” International Journal of Human-Computer Interaction, 31(12), pp. 882–889.
Kaarlela, T., Arnarson, H., Pitkäaho, T., Shu, B., Solvang, B., and Pieskä, S. (2022). “Common Educational Teleoperation Platform for Robotics Utilizing Digital Twins.” Machines, 10(7), p. 577.
Li, C., Fahmy, A., and Sienz, J. (2019). “An augmented reality based human-robot interaction interface using Kalman filter sensor fusion.” Sensors, 19(20), p. 4586.
Li, C., Zheng, P., Li, S., Pang, Y., and Lee, C. K. (2022). “AR-assisted digital twin-enabled robot collaborative manufacturing system with human-in-the-loop.” Robotics and Computer-Integrated Manufacturing, 76, p. 102321.
Melenbrink, N., Werfel, J., and Menges, A. (2020). “On-site autonomous construction robots: Towards unsupervised building.” Automation in construction, 119, p. 103312.
Moniruzzaman, M., Rassau, A., Chai, D., and Islam, S. M. S. (2022). “Teleoperation methods and enhancement techniques for mobile robots: A comprehensive survey.” Robotics and Autonomous Systems, 150, p. 103973.
Nenna, F., and Gamberini, L. “The influence of gaming experience, gender and other individual factors on robot teleoperations in vr.” Proc., 2022 17th ACM/IEEE International Conference on Human-Robot Interaction (HRI), IEEE, pp. 945–949.
Opiyo, S., Zhou, J., Mwangi, E., Kai, W., and Sunusi, I. (2021). “A review on teleoperation of mobile ground robots: Architecture and situation awareness.” International Journal of Control, Automation and Systems, 19, pp. 1384–1407.
Peppoloni, L., Brizzi, F., Avizzano, C. A., and Ruffaldi, E. “Immersive ROS-integrated framework for robot teleoperation.” Proc., 2015 IEEE Symposium on 3D User Interfaces (3DUI), IEEE, pp. 177–178.
Petereit, J., Beyerer, J., Asfour, T., Gentes, S., Hein, B., Hanebeck, U. D., Kirchner, F., Dillmann, R., Götting, H. H., and Weiser, M. “ROBDEKON: Robotic systems for decontamination in hazardous environments.” Proc., 2019 IEEE International Symposium on Safety, Security, and Rescue Robotics (SSRR), IEEE, pp. 249–255.
Peternel, L., Fang, C., Laghi, M., Bicchi, A., Tsagarakis, N., and Ajoudani, A. “Human arm posture optimisation in bilateral teleoperation through interface reconfiguration.” Proc., 2020 8th IEEE RAS/EMBS International Conference for Biomedical Robotics and Biomechatronics (BioRob), IEEE, pp. 1102–1108.
Roldán, J. J., Peña-Tapia, E., Martín-Barrio, A., Olivares-Méndez, M. A., Del Cerro, J., and Barrientos, A. (2017). “Multi-robot interfaces and operator situational awareness: Study of the impact of immersion and prediction.” Sensors, 17(8), p. 1720.
Schwarz, M., Rodehutskors, T., Droeschel, D., Beul, M., Schreiber, M., Araslanov, N., Ivanov, I., Lenz, C., Razlaw, J., and Schüller, S. (2017). “Nimbro rescue: solving disaster‐response tasks with the mobile manipulation robot momaro.” Journal of Field Robotics, 34(2), pp. 400–425.
Si, W., Wang, N., and Yang, C. (2021). “A review on manipulation skill acquisition through teleoperation‐based learning from demonstration.” Cognitive Computation and Systems, 3(1), pp. 1–16.
Siemens. (2019). “Siemens/Ros-sharp.” <https://github.com/siemens/ros-sharp>. (06 February, 2020).
Slater, M., Pérez Marcos, D., Ehrsson, H., and Sanchez-Vives, M. V. (2008). “Towards a digital body: the virtual arm illusion.” Frontiers in human neuroscience, p. 6.
Sober, S. J., and Sabes, P. N. (2005). “Flexible strategies for sensory integration during motor planning.” Nature neuroscience, 8(4), pp. 490–497.
Sreelakshmi, M., and Subash, T. (2017). “Haptic technology: A comprehensive review on its applications and future prospects.” Materials Today: Proceedings, 4(2), pp. 4182–4187.
Whitney, D., Rosen, E., Phillips, E., Konidaris, G., and Tellex, S. “Comparing robot grasping teleoperation across desktop and virtual reality with ROS reality.” Proc., Robotics Research: The 18th International Symposium ISRR, Springer, pp. 335–350.
wiki, R. (2019). “URDF.” <http://wiki.ros.org/urdf>. (06 February, 2020).
wiki, R. (2020). “ROS-bridge.” <http://wiki.ros.org/rosbridge_suite>. (06 February, 2020).
Xia, P., Xu, F., Song, Z., Li, S., and Du, J. (2023). “Sensory augmentation for subsea robot teleoperation.” Computers in Industry, 145, p. 103836.
Xia, P., You, H., Ye, Y., and Du, J. (2023). “ROV teleoperation via human body motion mapping: Design and experiment.” Computers in Industry, 150, p. 103959.
Xu, F., Xia, P., You, H., and Du, J. (2022). “Robotic Cross-Platform Sensor Fusion and Augmented Visualization for Large Indoor Space Reality Capture.” Journal of Computing in Civil Engineering, 36(6), p. 04022036.
Xu, F., Zhu, Q., Li, S., Song, Z., and Du, J. (2021). “VR-Based Haptic Simulator for Subsea Robot Teleoperations.” Computing in Civil Engineering 2021, pp. 1024–1032.
Ye, Y., Shi, Y., Srinivasan, D., and Du, J. (2022). “Sensation transfer for immersive exoskeleton motor training: Implications of haptics and viewpoints.” Automation in Construction, 141, p. 104411.
Ye, Y., Xia, P., Zhou, T., and Du, J. (2023). “Spatial Memory of BIM and Virtual Reality: Mental Mapping Study.” Journal of Construction Engineering and Management, 149(7), p. 04023042.
Ye, Y., You, H., and Du, J. (2023). “Improved Trust in Human-Robot Collaboration with ChatGPT.”.
Ye, Y., Zhou, T., and Du, J. (2023). “Robot-Assisted Immersive Kinematic Experience Transfer for Welding Training.” Journal of Computing in Civil Engineering, 37(2), p. 04023002.
Zhou, T., Zhu, Q., and Du, J. (2020). “Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction.” Advanced Engineering Informatics, 46.
Zhou, T., Zhu, Q., and Du, J. (2020). “Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction.” Advanced Engineering Informatics, 46, p. 101170.
Zhou, T., Zhu, Q., Shi, Y., and Du, J. (2022). “Construction robot teleoperation safeguard based on real-time human hand motion prediction.” Journal of Construction Engineering and Management, 148(7), p. 04022040.
Zhou, T., Zhu, Q., Ye, Y., and Du, J. (2023). “Humanlike Inverse Kinematics for Improved Spatial Awareness in Construction Robot Teleoperation: Design and Experiment.” Journal of Construction Engineering and Management, 149(7), p. 04023044.
Zhu, Q., Zhou, T., and Du, J. (2022). “Haptics-based force balance controller for tower crane payload sway controls.” Automation in Construction, 144, p. 104597.
Information & Authors
Information
Published In
Construction Research Congress 2024
Pages: 727 - 738
History
Published online: Mar 18, 2024
ASCE Technical Topics:
- Analysis (by type)
- Automation and robotics
- Business management
- Construction engineering
- Construction industry
- Construction management
- Engineering fundamentals
- Engines
- Equipment and machinery
- Human and behavioral factors
- Material mechanics
- Material properties
- Materials engineering
- Practice and Profession
- Resilient modulus
- Spatial analysis
- Spatial data
- Systems engineering
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.