Motion-Based Control Interface for Intuitive and Efficient Teleoperation of Construction Robots
Publication: Computing in Civil Engineering 2023
ABSTRACT
Robotic teleoperation is regarded as a promising method of assisting workers in performing construction activities while avoiding potentially dangerous conditions. However, the control interface of conventional teleoperation systems relies on physical devices, such as a keyboard and joystick, to input control commands, which cannot be intuitively mapped to robot movement, and thus affecting the teleoperation performance of non-expert users. In this research, we present a novel human-robot interface to map human arm movement to robotic arm motion for intuitive teleoperation. Firstly, the human arm is tracked in real-time through a motion capturing system, that is, OptiTrack. The 3D location data is then streamed to robot operating system (ROS) and is used for motion planning of the robotic arm in MoveIt. Finally, the interface is demonstrated in a teleoperated pick-and-place task through robotic simulation, and the usability of the developed framework is compared to that of a joystick-based teleoperation system.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Al-Mouhamed, M. A., M. Nazeeruddin, and N. Merah. (2008). “Design and instrumentation of force feedback in telerobotics.” IEEE Transactions on Instrumentation and Measurement 58(6): 1949–1957.
Bao, S.-D., X.-L. Meng, W. Xiao, and Z.-Q. Zhang. (2017). “Fusion of inertial/magnetic sensor measurements and map information for pedestrian tracking.” Sensors 17(2): 340.
Bertomeu-Motos, A., L. D. Lledó, J. A. Díez, J. M. Catalan, S. Ezquerro, F. J. Badesa, and N. Garcia-Aracil. (2015). “Estimation of human arm joints using two wireless sensors in robotic rehabilitation tasks.” Sensors 15(12): 30571–30583.
Czarnowski, J., A. Dąbrowski, M. Maciaś, J. Główka, and J. Wrona. (2018). “Technology gaps in human-machine interfaces for autonomous construction robots.” Automation in Construction 94: 179–190.
Du, G., and P. Zhang. (2014). “A markerless human–robot interface using particle filter and Kalman filter for dual robots.” IEEE Transactions on Industrial Electronics 62(4): 2257–2264.
Fang, B., F. Sun, H. Liu, D. Guo, W. Chen, and G. Yao. (2017). “Robotic teleoperation systems using a wearable multimodal fusion device.” International journal of advanced robotic systems 14(4): 1729881417717057.
Galzarano, S., R. Giannantonio, A. Liotta, and G. Fortino. (2014). “A task-oriented framework for networked wearable computing.” IEEE Transactions on Automation Science and Engineering 13(2): 621–638.
Gromov, B., G. Abbate, L. M. Gambardella, and A. Giusti. (2019). “Proximity human-robot interaction using pointing gestures and a wrist-mounted IMU.” International Conference on Robotics and Automation (ICRA), IEEE pp. 8084–8091.
Hasegawa, Y., T. Kitamura, S. Sakaino, and T. Tsuji. (2020). “Bilateral control of elbow and shoulder joints using functional electrical stimulation between humans and robots.” IEEE Access 8: 15792–15799.
Hirche, S., and M. Buss. (2012). “Human-oriented control for haptic teleoperation.” Proceedings of the IEEE 100(3): 623–647.
Kim, D., J. Kim, K. Lee, C. Park, J. Song, and D. Kang. (2009). “Excavator tele-operation system using a human arm.” Automation in construction 18(2): 173–182.
Köpüklü, O., A. Gunduz, N. Kose, and G. Rigoll. (2019). “Real-time hand gesture detection and classification using convolutional neural networks.” 14th IEEE international conference on automatic face & gesture recognition (FG 2019), IEEE.
Lu, Z., P. Huang, and Z. Liu. (2017). “Predictive approach for sensorless bimanual teleoperation under random time delays with adaptive fuzzy control.” IEEE Transactions on Industrial Electronics 65(3): 2439–2448.
Macchini, M., J. Frogg, F. Schiano, and D. Floreano. (2022). “Does spontaneous motion lead to intuitive Body-Machine Interfaces? A fitness study of different body segments for wearable telerobotics.” 31st IEEE International Conference on Robot and Human Interactive Communication (RO-MAN), IEEE.
Mavridis, N., G. Pierris, P. Gallina, Z. Papamitsiou, and U. Saad. (2015). “On the subjective difficulty of joystick-based robot arm teleoperation with auditory feedback.” IEEE 8th GCC Conference & Exhibition, IEEE.
Molchanov, P., X. Yang, S. Gupta, K. Kim, S. Tyree, and J. Kautz. (2016). “Online detection and classification of dynamic hand gestures with recurrent 3d convolutional neural network.” Proceedings of the IEEE conference on computer vision and pattern recognition.
Moschetti, A., F. Cavallo, D. Esposito, J. Penders, and A. Di Nuovo. (2019). “Wearable sensors for human–robot walking together.” Robotics 8(2): 38.
OptiTrack. (2020). “Motive Documentation - NaturalPoint Product Documentation,.” Retrieved 2023, Februrary 09, from https://v22.wiki.optitrack.com/index.php?title=Motive_Documentation.
OptiTrack. (2022). “Rigid Body Tracking.” Retrieved 2023, Februrary 09, from https://docs.optitrack.com/v/v2.3/motive/rigid-body-tracking.
Repnik, E., U. Puh, N. Goljar, M. Munih, and M. Mihelj. (2018). “Using inertial measurement units and electromyography to quantify movement during action research arm test execution.” Sensors 18(9): 2767.
Rudigkeit, N., and M. Gebhard. (2019). “AMiCUS—A head motion-based interface for control of an assistive robot.” Sensors 19(12): 2836.
Škulj, G., R. Vrabič, and P. Podržaj. (2021). “A wearable imu system for flexible teleoperation of a collaborative industrial robot.” Sensors 21(17): 5871.
Statistics, U. s. b. o. l. (2022). “OSHA Data & Statistics Commonly Used Statistics.” Retrieved 2023, March 10, from https://www.bls.gov/news.release/cfoi.t03.htm.
Taylor, R. M., T. C. Hudson, A. Seeger, H. Weber, J. Juliano, and A. T. Helser. (2001). “VRPN: a device-independent, network-transparent VR peripheral system.” Proceedings of the ACM symposium on Virtual reality software and technology.
Truong, D. Q., B. N. M. Truong, N. T. Trung, S. A. Nahian, and K. K. Ahn. (2017). “Force reflecting joystick control for applications to bilateral teleoperation in construction machinery.” International Journal of Precision Engineering and Manufacturing 18: 301–315.
Van der Kruk, E., and M. M. Reijne. (2018). “Accuracy of human motion capture systems for sport applications; state-of-the-art review.” European journal of sport science 18(6): 806–819.
Wang, X., and Z. Zhu. (2021). “Vision–based framework for automatic interpretation of construction workers’ hand gestures.” Automation in Construction 130: 103872.
Xu, C., J. He, X. Zhang, X. Zhou, and S. Duan. (2019). “Towards human motion tracking: Multi-sensory IMU/TOA fusion method and fundamental limits.” Electronics 8(2): 142.
Zhang, M., R. Xu, H. Wu, J. Pan, and X. Luo. (2023). “Human–robot collaboration for on-site construction.” Automation in Construction 150: 104812.
Zhou, H., and H. Hu. (2008). “Human motion tracking for rehabilitation—A survey.” Biomedical signal processing and control 3(1): 1–18.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 470 - 478
History
Published online: Jan 25, 2024
ASCE Technical Topics:
- Automation and robotics
- Business management
- Construction engineering
- Construction methods
- Continuum mechanics
- Dynamics (solid mechanics)
- Employment
- Engineering fundamentals
- Engineering mechanics
- Geomatics
- Human and behavioral factors
- Hydrologic data
- Hydrologic engineering
- Hydrology
- Labor
- Mapping
- Motion (dynamics)
- Personnel management
- Practice and Profession
- Solid mechanics
- Surveying methods
- Systems engineering
- Water and water resources
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.