Humanlike Inverse Kinematics for Improved Spatial Awareness in Construction Robot Teleoperation: Design and Experiment
Publication: Journal of Construction Engineering and Management
Volume 149, Issue 7
Abstract
The teleoperation of robotic arms is expected to play a key role in dangerous or inaccessible construction workplaces. Most robot arms rely on mechanical designs that are completely different from human arms. It could lead to a risk that certain joints move with undesired poses and cause collisions because of the mismatch between robot mechanical design and human operators’ egocentric perception of their own arms. This paper proposes an innovative robotic control method that mimics the human shoulder-arm structure, enabling human operators to teleoperate with unfamiliar robotic arms intuitively. A two-tracker system is used to map human arm motions into a revised robotic inverse kinematics (IK) algorithm called humanlike IK. One tracker controls the end effector of the robot, and the other tracker drives the middle joint of the robot, analogous to how a human moves the arm. A seven-degree-of-freedom robotic arm was repurposed based on the revised IK. A human-subject experiment () was performed to test the effectiveness of the proposed humanlike IK method in a pipe maintenance task. Results confirmed the performance and functional benefits of the proposed method. It can inspire the design of a new robot teleoperation method for dexterous tasks in construction.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data are available upon request. The video link of the real and virtual robots being controlled in the same pose can be found at https://youtu.be/d6ainvm6sk0. The raw data collected by the system on participants’ hand movement, elbow movement, robot joint movement, task complete states, and collision times at a frequency of 90 Hz can be found at https://www.dropbox.com/sh/jbokwdr06wbwecb/AABIdESNJ5WzRodvKl0Zut4ra?dl=0.
Acknowledgments
This material is supported by the National Science Foundation (NSF) under Grant Nos. 1937053 and 2024784. Any opinions, findings, conclusions, or recommendations expressed in this article are those of the authors and do not reflect the views of the NSF.
References
Anderson, R. J., and M. W. Spong. 1988. “Bilateral control of teleoperators with time delay.” In Proc., 1988 IEEE Int. Conf. on Systems, Man, and Cybernetics, 131–138. New York: IEEE.
Aristidou, A., J. Lasenby, Y. Chrysanthou, and A. Shamir. 2018. “Inverse kinematics techniques in computer graphics: A survey.” In Proc., Computer Graphics Forum, 35–58. New York: Wiley.
Barsalou, L. 2003. “Situated simulation in the human conceptual system.” Lang. Cognit. Processes 18 (5–6): 513–562. https://doi.org/10.1080/01690960344000026.
Barsalou, L. W. 1999. “Perceptual symbol systems.” Behav. Brain Sci. 22 (4): 577–660. https://doi.org/10.1017/S0140525X99002149.
Barsalou, L. W. 2008. “Grounded cognition.” Annu. Rev. Psychol. 59 (5): 617–645. https://doi.org/10.1146/annurev.psych.59.103006.093639.
Bennett, D., J. Hollerbach, Y. Xu, and I. Hunter. 1992. “Time-varying stiffness of human elbow joint during cyclic voluntary movement.” Exp. Brain Res. 88 (2): 433–442. https://doi.org/10.1007/BF02259118.
Bonci, A., P. D. Cen Cheng, M. Indri, G. Nabissi, and F. Sibona. 2021. “Human-robot perception in industrial environments: A survey.” Sensors 21 (5): 1571. https://doi.org/10.3390/s21051571.
Buckley, M., A. Yardley, G. Johnson, and D. Cams. 1996. “Dynamics of the upper limb during performance of the tasks of everyday living—A review of the current knowledge base.” Proc. Inst. Mech. Eng., Part H: J. Eng. Med. 210 (4): 241–247. https://doi.org/10.1243/PIME_PROC_1996_210_420_02.
Buss, S. R. 2004. “Introduction to inverse kinematics with Jacobian transpose, pseudoinverse and damped least squares methods.” IEEE J. Rob. Autom. 17 (1–19): 16.
Cantrell, C. D. 2000. Modern mathematical methods for physicists and engineers. Cambridge, UK: Cambridge University Press.
Canutescu, A. A., and R. L. Dunbrack Jr. 2003. “Cyclic coordinate descent: A robotics algorithm for protein loop closure.” Protein Sci. 12 (5): 963–972. https://doi.org/10.1110/ps.0242703.
Casalino, A., C. Messeri, M. Pozzi, A. M. Zanchettin, P. Rocco, and D. Prattichizzo. 2018. “Operator awareness in human–Robot collaboration through wearable vibrotactile feedback.” IEEE Rob. Autom. Lett. 3 (4): 4289–4296. https://doi.org/10.1109/LRA.2018.2865034.
Cavallin, K., and P. Svensson. 2009. Semi-Autonomous, teleoperated search and rescue robot. Uppsala, Sweden: Uppsala Univ.
Chen, J. Y., E. C. Haas, and M. J. Barnes. 2007. “Human performance issues and user interface design for teleoperated robots.” IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 37 (6): 1231–1245. https://doi.org/10.1109/TSMCC.2007.905819.
Chitta, S. 2016. “Moveit: An introduction.” In Robot operating system (ROS), 3–27. Berlin: Springer.
Chitta, S., I. Sucan, and S. Cousins. 2012. “Moveit [ROS topics].” IEEE Rob. Autom. Mag. 19 (1): 18–19. https://doi.org/10.1109/MRA.2011.2181749.
Craig, J. J. 2009. Introduction to robotics: Mechanics and control, 3/E. Noida, India: Pearson Education.
Crick, C., G. Jay, S. Osentoski, B. Pitzer, and O. C. Jenkins. 2017. “Rosbridge: ROS for non-ROS users.” In Robotics research, 493–504. Berlin: Springer.
Diankov, R. 2010. “Ikfast: The robot kinematics compiler.” Accessed May 23, 2022. http://openrave.org/docs/latest_stable/openravepy/ikfast/#ikfast-the-robot-kinematics-compiler.
Doisy, G., A. Ronen, and Y. Edan. 2017. “Comparison of three different techniques for camera and motion control of a teleoperated robot.” Appl. Ergon. 58 (5): 527–534. https://doi.org/10.1016/j.apergo.2016.05.001.
Drury, J. L., J. Scholtz, and H. A. Yanco. 2003. “Awareness in human-robot interactions.” In Proc., SMC’03 Conf. Proc. 2003 IEEE Int. Conf. on Systems, Man and Cybernetics. Conf. Theme-System Security and Assurance (Cat. No. 03CH37483), 912–918. New York: IEEE.
Du, J., Y. Shi, C. Mei, J. Quarles, and W. Yan. 2016. “Communication by interaction: A multiplayer VR environment for building walkthroughs.” In Proc., Construction Research Congress 2016, 2281–2290. Reston, VA: ASCE.
Du, J., Y. Shi, Z. Zou, and D. Zhao. 2018a. “CoVR: Cloud-based multiuser virtual reality headset system for project communication of remote users.” J. Constr. Eng. Manage. 144 (2): 04017109. https://doi.org/10.1061/(ASCE)CO.1943-7862.0001426.
Du, J., Z. Zou, Y. Shi, and D. Zhao. 2017. “Simultaneous data exchange between BIM and VR for collaborative decision making.” Comput. Civ. Eng. 2017 (Jun): 1–8.
Du, J., Z. Zou, Y. Shi, and D. Zhao. 2018b. “Zero latency: Real-time synchronization of BIM data in virtual reality for collaborative decision-making.” Autom. Constr. 85 (Jan): 51–64. https://doi.org/10.1016/j.autcon.2017.10.009.
El-Sherbiny, A., M. A. Elhosseini, and A. Y. Haikal. 2018. “A comparative study of soft computing methods to solve inverse kinematics problem.” Ain Shams Eng. J. 9 (4): 2535–2548. https://doi.org/10.1016/j.asej.2017.08.001.
Emika, F. 2022. “Franka control interface documentation–Franka Control Interface (FCI) documentation.” Accessed May 23, 2022. https://frankaemika.github.io/docs/.
Freina, L., and M. Ott. 2015. “A literature review on immersive virtual reality in education: State of the art and perspectives.” In Proc., Int. Scientific Conf. Elearning and Software for Education, 1101. Bucharest, Romania: Romanian Advanced Distributed Learning Association.
Gan, J. Q., E. Oyama, E. M. Rosales, and H. Hu. 2005. “A complete analytical solution to the inverse kinematics of the Pioneer 2 robotic arm.” Robotica 23 (1): 123–129. https://doi.org/10.1017/S0263574704000529.
GitHub. 2019. “ROS# GitHub page.” Accessed February 6, 2020. https://github.com/siemens/ros-sharp.
Hart, S. G., and L. E. Staveland. 1988. “Development of NASA-TLX (Task Load Index): Results of empirical and theoretical research.” In Advances in psychology, 139–183. Amsterdam, Netherlands: Elsevier.
Hirche, S., and M. Buss. 2012. “Human-oriented control for haptic teleoperation.” Proc. IEEE 100 (3): 623–647.
Hitz, G., A. Gotovos, M.-É. Garneau, C. Pradalier, A. Krause, and R. Y. Siegwart. 2014. “Fully autonomous focused exploration for robotic environmental monitoring.” In Proc., 2014 IEEE Int. Conf. on Robotics and Automation (ICRA), 2658–2664. New York: IEEE.
Ikuta, K., K. Yamamoto, and K. Sasaki. 2003. “Development of remote microsurgery robot and new surgical procedure for deep and narrow space.” In Proc., 2003 IEEE Int. Conf. on Robotics and Automation (Cat. No. 03CH37422), 1103–1108. New York: IEEE.
Ishida, R., L. Meli, Y. Tanaka, K. Minamizawa, and D. Prattichizzo. 2018. “Sensory-motor augmentation of the robot with shared human perception.” In Proc., 2018 IEEE/RSJ Int. Conf. on Intelligent Robots and Systems (IROS), New York: IEEE.
Jiang, H., J. P. Wachs, M. Pendergast, and B. S. Duerstock. 2013. “3D joystick for robotic arm control by individuals with high level spinal cord injuries.” In Proc., 2013 IEEE 13th Int. Conf. on Rehabilitation Robotics (ICORR), 1–5. New York: IEEE.
Kanehiro, F., H. Hirukawa, K. Kaneko, S. Kajita, K. Fujiwara, K. Harada, and K. Yokoi. 2004. “Locomotion planning of humanoid robots to pass through narrow spaces.” In Proc., IEEE Int. Conf. on Robotics and Automation, 2004, 604–609. New York: IEEE.
Kenwright, B. 2012. “Inverse kinematics–cyclic coordinate descent (CCD).” J. Graphics Tools 16 (4): 177–217. https://doi.org/10.1080/2165347X.2013.823362.
Kim, D., M. Liu, S. Lee, and V. R. Kamat. 2019. “Trajectory prediction of mobile construction resources toward pro-active struck-by hazard detection.” In Proc., Int. Symp. on Automation and Robotics in Construction (IAARC). Banff, Canada: IAARC.
Kinateder, M., E. Ronchi, D. Nilsson, M. Kobes, M. Müller, P. Pauli, and A. Mühlberger. 2014. “Virtual reality for fire evacuation research.” In Proc., 2014 Federated Conf. on Computer Science and Information Systems, 313–321. New York: IEEE.
Kohn, S. C., E. J. De Visser, E. Wiese, Y.-C. Lee, and T. H. Shaw. 2021. “Measurement of trust in automation: A narrative review and reference guide.” Front. Psychol. 12 (Oct): 604977. https://doi.org/10.3389/fpsyg.2021.604977.
Lander, J., and G. Content. 1998. “Making kine more flexible.” Game Developer Mag. 1 (15–22): 2.
Lenzi, T., N. Vitiello, S. M. M. De Rossi, S. Roccella, F. Vecchi, and M. C. Carrozza. 2011. “NEUROExos: A variable impedance powered elbow exoskeleton.” In Proc., 2011 IEEE Int. Conf. on Robotics and Automation, 1419–1426. New York: IEEE.
Li, X., S. Liu, Y. Chang, S. Li, Y. Fan, and H. Yu. 2020. “A human joint torque estimation method for elbow exoskeleton control.” Int. J. Humanoid Rob. 17 (3): 1950039. https://doi.org/10.1142/S0219843619500397.
Luenberger, D. G., and Y. Ye. 1984. Linear and nonlinear programming. Berlin: Springer.
Manocha, D., and J. F. Canny. 1994. “Efficient inverse kinematics for general 6R manipulators.” IEEE Trans. Rob. Autom. 10 (5): 648–657. https://doi.org/10.1109/70.326569.
Mathew, E. B., D. Khanduja, B. Sapra, and B. Bhushan. 2015. “Robotic arm control through human arm movement detection using potentiometers.” In Proc., 2015 Int. Conf. on Recent Developments in Control, Automation and Power Engineering (RDCAPE), 298–303. New York: IEEE.
Merritt, S. M. 2011. “Affective processes in human–automation interactions.” Hum. Factors 53 (4): 356–370. https://doi.org/10.1177/0018720811411912.
Murphy, R. R. 2004. “Human–robot interaction in rescue robotics.” IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 34 (2): 138–153. https://doi.org/10.1109/TSMCC.2004.826267.
Niemeyer, G., C. Preusche, S. Stramigioli, and D. Lee. 2016. “Telerobotics.” In Springer handbook of robotics, 1085–1108. Berlin: Springer.
Opiyo, S., J. Zhou, E. Mwangi, W. Kai, and I. Sunusi. 2021. “A review on teleoperation of mobile ground robots: Architecture and situation awareness.” Int. J. Control Autom. Syst. 19 (3): 1384–1407. https://doi.org/10.1007/s12555-019-0999-z.
Osawa, H., A. Ema, H. Hattori, N. Akiya, N. Kanzaki, A. Kubo, T. Koyama, and R. Ichise. 2017. “Analysis of robot hotel: Reconstruction of works with robots.” In Proc., 2017 26th IEEE Int. Symp. on Robot and Human Interactive Communication (RO-MAN), 219–223. New York: IEEE.
Paul, R. P., and B. Shimano. 1979. “Kinematic control equations for simple manipulators.” In Proc., IEEE Conf. on Decision and Control including the 17th Symp. on Adaptive Processes, 1398–1406. New York: IEEE.
Penrose, R. 1955. “A generalized inverse for matrices.” In Proc., Mathematical Cambridge Philosophical Society, 406–413. Cambridge, UK: Cambridge University Press.
Peternel, L., N. Tsagarakis, and A. Ajoudani. 2017. “A human–robot co-manipulation approach based on human sensorimotor information.” IEEE Trans. Neural Syst. Rehabil. Eng. 25 (7): 811–822. https://doi.org/10.1109/TNSRE.2017.2694553.
Pincus, S. M., and R. R. Viscarello. 1992. “Approximate entropy: A regularity measure for fetal heart rate analysis.” Obstet. Gynecol. 79 (2): 249–255.
Pu, G., P. Wei, A. Aribe, J. Boultinghouse, N. Dinh, F. Xu, and J. Du. 2021. “Seeing through walls: Real-time digital twin modeling of indoor spaces.” In Proc., 2021 Winter Simulation Conf. (WSC), 1–10. New York: IEEE.
Qian, K., A. Song, J. Bao, and H. Zhang. 2012. “Small teleoperated robot for nuclear radiation and chemical leak detection.” Int. J. Adv. Rob. Syst. 9 (3): 70. https://doi.org/10.5772/50720.
Qin, Z., P. Wang, J. Sun, J. Lu, and H. Qiao. 2016. “Precise robotic assembly for large-scale objects based on automatic guidance and alignment.” IEEE Trans. Instrum. Meas. 65 (6): 1398–1411. https://doi.org/10.1109/TIM.2016.2526738.
Quigley, M., K. Conley, B. Gerkey, J. Faust, T. Foote, J. Leibs, R. Wheeler, and A. Y. Ng. 2009. “ROS: An open-source robot operating system.” In Proc., ICRA workshop on open source software. New York: IEEE.
Raghavan, M., and B. Roth. 1993. Inverse kinematics of the general 6R manipulator and related linkages. New York: ASME.
Rakita, D., B. Mutlu, and M. Gleicher. 2018. “An autonomous dynamic camera method for effective remote teleoperation.” In Proc., 2018 ACM/IEEE Int. Conf. on Human-Robot Interaction, 325–333. New York: IEEE.
Reddivari, H., C. Yang, Z. Ju, P. Liang, Z. Li, and B. Xu. 2014. “Teleoperation control of Baxter robot using body motion tracking.” In Proc., 2014 Int. Conf. on Multisensor Fusion and Information Integration for Intelligent systems (MFI), 1–6. New York: IEEE.
Richman, J. S., and J. R. Moorman. 2000. “Physiological time-series analysis using approximate entropy and sample entropy.” Am. J. Physiol. Heart Circle Physiol. 2000 (Jun): 2039–2049. https://doi.org/10.1152/ajpheart.2000.278.6.H2039.
Scholtz, J. 2003. “Theory and evaluation of human robot interactions.” In Proc., 36th Annual Hawaii Int. Conf. on System Sciences. New York: IEEE.
Sciavicco, L., and B. Siciliano. 2012. Modelling and control of robot manipulators. Berlin: Springer.
Sheridan, T. B. 1992. Telerobotics, automation, and human supervisory control. Cambridge, MA: MIT Press.
Shi, Y., J. Du, E. Ragan, K. Choi, and S. Ma. 2018. “Social influence on construction safety behaviors: A multi-user virtual reality experiment.” In Proc., Construction Research Congress, 147–183. Reston, VA: ASCE.
Shou, W., J. Wang, and X. Wang. 2018. “4D BIM for improving plant turnaround maintenance planning and execution: A case study.” In Proc., ISARC. Int. Conf. Symp. on Automation and Robotics in Construction, 1–8. Waterloo, ON: IAARC Publications.
Shukla, A., and H. Karki. 2016. “Application of robotics in offshore oil and gas industry—A review Part II.” Rob. Auton. Syst. 75 (Jun): 508–524. https://doi.org/10.1016/j.robot.2015.09.013.
Siciliano, B., and O. Khatib. 2016. Springer handbook of robotics. Berlin: Springer.
Siemens. 2019. “Siemens/ROS-sharp.” Accessed February 6, 2020. https://github.com/siemens/ros-sharp.
Stanton, C., A. Bogdanovych, and E. Ratanasena. 2012. “Teleoperation of a humanoid robot using full-body motion capture, example movements, and machine learning.” In Proc., Australasian Conf. on Robotics and Automation, 51. Melbourne, VIC, Australia: Australian Robotics and Automation Association.
Taylor, R. M. 2017. “Situational awareness rating technique (SART): The development of a tool for aircrew systems design.” In Situational awareness, 111–128. Abingdon, UK: Routledge.
Thirumurugan, J., M. Vinoth, G. Kartheeswaran, and M. Vishwanathan. 2010. “Line following robot for library inventory management system.” In Proc., INTERACT-2010, 1–3. New York: IEEE.
van Osch, M., D. Bera, K. van Hee, Y. Koks, and H. Zeegers. 2014. “Tele-operated service robots: ROSE.” Autom. Constr. 39 (6): 152–160. https://doi.org/10.1016/j.autcon.2013.06.009.
Vu, M. T., H.-S. Choi, N. D. Nguyen, and S.-K. Kim. 2019. “Analytical design of an underwater construction robot on the slope with an up-cutting mode operation of a cutter bar.” Appl. Ocean Res. 86 (Jun): 289–309. https://doi.org/10.1016/j.apor.2019.02.019.
Wang, L.-C., and C.-C. Chen. 1991. “A combined optimization method for solving the inverse kinematics problems of mechanical manipulators.” IEEE Trans. Rob. Autom. 7 (4): 489–499. https://doi.org/10.1109/70.86079.
Welman, C. 1993. “Inverse kinematics and geometric constraints for articulated figure manipulation.” Ph.D. thesis, School of Computing Science, Simon Fraser Univ.
Wiki, R. 2019. “URDF.” Accessed February 6, 2020. http://wiki.ros.org/urdf.
Wilson, M. 2002. “Six views of embodied cognition.” Psychonomic Bull. Rev. 9 (4): 625–636. https://doi.org/10.3758/BF03196322.
Xia, P., F. Xu, Q. Zhu, and J. Du. 2022. “Human robot comparison in rapid structural inspection.” In Proc., Construction Research Congress 2022, 570–580. Reston, VA: ASCE.
Yanco, H. A., and J. Drury. 2004. “Classifying human-robot interaction: An updated taxonomy.” In Proc., 2004 IEEE Int. Conf. on Systems, Man and Cybernetics (IEEE Cat. No. 04CH37583), 2841–2846. New York: IEEE.
Yanco, H. A., and J. L. Drury. 2002. “A taxonomy for human-robot interaction.” In Proc., AAAI Fall Symp. on Human-Robot Interaction, 111-119. Palo Alto, CA: Association for the Advancement of Artificial Intelligence.
Yang, C., X. Wang, L. Cheng, and H. Ma. 2016. “Neural-learning-based telerobot control with guaranteed performance.” IEEE Trans. Cybern. 47 (10): 3148–3159. https://doi.org/10.1109/TCYB.2016.2573837.
Yentes, J. M., N. Hunt, K. K. Schmid, J. P. Kaipust, D. McGrath, and N. Stergiou. 2013. “The appropriate use of approximate entropy and sample entropy with short data sets.” Ann. Biomed. Eng. 41 (2): 349–365. https://doi.org/10.1007/s10439-012-0668-3.
Yokokohji, Y., and T. Yoshikawa. 1994. “Bilateral control of master-slave manipulators for ideal kinesthetic coupling-formulation and experiment.” IEEE Trans. Rob. Autom. 10 (5): 605–620. https://doi.org/10.1109/70.326566.
Zhou, T., Y. Wang, Q. Zhu, and J. Du. 2022. “Human hand motion prediction based on feature grouping and deep learning: Pipe skid maintenance example.” Autom. Constr. 138 (Jun): 104232. https://doi.org/10.1016/j.autcon.2022.104232.
Zhou, T., Q. Zhu, and J. Du. 2020. “Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction.” Adv. Eng. Inf. 46 (Oct): 101170. https://doi.org/10.1016/j.aei.2020.101170.
Zhu, Q., J. Du, Y. Shi, and P. Wei. 2021. “Neurobehavioral assessment of force feedback simulation in industrial robotic teleoperation.” Autom. Constr. 126 (Jun): 103674. https://doi.org/10.1016/j.autcon.2021.103674.
Zhu, Q., T. Zhou, and J. Du. 2022. “Upper-body haptic system for snake robot teleoperation in pipelines.” Adv. Eng. Inf. 51 (Jan): 101532. https://doi.org/10.1016/j.aei.2022.101532.
Information & Authors
Information
Published In
Journal of Construction Engineering and Management
Volume 149 • Issue 7 • July 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Nov 16, 2022
Accepted: Feb 15, 2023
Published online: Apr 22, 2023
Published in print: Jul 1, 2023
Discussion open until: Sep 22, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Yan Li, Songyang Liu, Mengjun Wang, Shuai Li, Jindong Tan, Teleoperation-Driven and Keyframe-Based Generalizable Imitation Learning for Construction Robots, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5884, 38, 6, (2024).
- Sungboo Yoon, Moonseo Park, Changbum R. Ahn, LaserDex: Improvising Spatial Tasks Using Deictic Gestures and Laser Pointing for Human–Robot Collaboration in Construction, Journal of Computing in Civil Engineering, 10.1061/JCCEE5.CPENG-5715, 38, 3, (2024).
- Yang Ye, Thanakon Uthai, Pengxiang Xia, Tianyu Zhou, Jing Du, User Experience and Workload Evaluation in Robot-Assisted Virtual Reality Welding Training, Construction Research Congress 2024, 10.1061/9780784485293.011, (99-108), (2024).
- Fang Xu, Tianyu Zhou, Yang Ye, Jing Du, Augmented Telepresence: Enhancing Robot Arm Control with Mixed Reality for Dexterous Manipulation, Construction Research Congress 2024, 10.1061/9780784485262.074, (727-738), (2024).