Human Body Motion and Hand Gesture Control for Remotely Operated Vehicle (ROV) Teleoperation
Publication: Computing in Civil Engineering 2023
ABSTRACT
Remotely operated vehicles (ROVs) are widely used in subsea navigation and operation tasks. Currently, ROV operations are primarily conducted through traditional control kiosks and feedback methods, with the camera display and joysticks control on a surface vessel. This kind of control method resulted in an extremely high training barrier. The marine and offshore sectors are dominated by male workers due to the harsh working conditions. Besides, subsea engineering also poses strict requirements on age, because of the mental and physical load of certain tasks. Therefore, this paper proposes a human body motion and hand gesture control method in virtual reality (VR) for ROV teleoperation, aiming to lower the training barrier, improve the work condition, and reduce mental load during the tasks. Specifically, human body motion parameters, mainly motion and orientation, are recorded in VR and mapped with ROV locomotion control via closed-loop control. A series of scripts are developed to recognize human gestures as functional control, such as light on/off or ROV up/down control. Finally, a configuration UI panel is designed for personnel adjustment. A case study was also presented to verify the effectiveness of the system. With this control system, the operator could teleoperate ROV in a more comfortable and safer environment in a more intuitive manner.
Get full access to this article
View all available purchase options and get full access to this chapter.
REFERENCES
Amemiya, T., and Maeda, T. (2009). “Directional force sensation by asymmetric oscillation from a double-layer slider-crank mechanism.” Journal of computing and information science in engineering, 9(1).
Baird, S. L. (2005). “Deep-Sea Exploration: Earth’s Final Frontier.” Technology and Engineering Teacher, 65(4), 14.
bHaptics. (2022). “TactSuit X40.” <https://www.bhaptics.com/tactsuit/tactsuit-x40>. (July 8th, 2022).
Casey, J. (2020). “Drawing the line: could the subsea industry turn away from oil and gas.” <https://www.offshore-technology.com/analysis/drawing-the-line-could-the-subsea-industry-turn-away-from-oil-and-gas/>. (2022).
Ciriello, V. M., Maikala, R. V., and O’Brien, N. V. (2013). “Maximal acceptable torques of six highly repetitive hand-wrist motions for male industrial workers.” Human factors, 55(2), 309–322.
Concannon, D., Flynn, R., and Murray, N. “A quality of experience evaluation system and research challenges for networked virtual reality-based teleoperation applications.” Proc., Proceedings of the 11th ACM workshop on immersive mixed and virtual environment systems, 10–12.
Du, J., Shi, Y., Zou, Z., and Zhao, D. (2018). “CoVR: Cloud-based multiuser virtual reality headset system for project communication of remote users.” Journal of Construction Engineering and Management, 144(2), 04017109.
Du, J., Zou, Z., Shi, Y., and Zhao, D. (2018). “Zero latency: Real-time synchronization of BIM data in virtual reality for collaborative decision-making.” Automation in Construction, 85, 51–64.
Elor, A., Thang, T., Hughes, B. P., Crosby, A., Phung, A., Gonzalez, E., Katija, K., Haddock, S. H., Martin, E. J., and Erwin, B. E. “Catching Jellies in Immersive Virtual Reality: A Comparative Teleoperation Study of ROVs in Underwater Capture Tasks.” Proc., Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology, 1–10.
Finney, G. R. (2015). “Perceptual-motor dysfunction.” Continuum: Lifelong Learning in Neurology, 21(3), 678–689.
Harmonic, W. (2022). “Crest Ocean System HDRP.” <https://assetstore.unity.com/packages/tools/particles-effects/crest-ocean-system-hdrp-164158#description>. (July 5th, 2022).
Index, L. (1990). Results of empirical and theoretical research. Advances in.
Kennedy, B. R., Cantwell, K., Malik, M., Kelley, C., Potter, J., Elliott, K., Lobecker, E., Gray, L. M., Sowers, D., and White, M. P. (2019). “The unknown and the unexplored: Insights into the Pacific deep-sea following NOAA CAPSTONE expeditions.” Frontiers in Marine Science, 6, 480.
Lawrance, N., and Hollinger, G. A. (2018). Industry Feedback: Opportunities for Autonomous Monitoring and Intervention in Marine Renewable Energy Arrays.
Li, X., Chen, G., Chang, Y., and Xu, C. (2019). “Risk-based operation safety analysis during maintenance activities of subsea pipelines.” Process Safety and Environmental Protection, 122, 247–262.
Li, Y., Lv, C., and Xue, J. “A novel predictive haptic control interface for automation-to-human takeover of automated vehicles.” Proc., 2019 IEEE Intelligent Vehicles Symposium (IV), IEEE, 994–999.
Lwin, K. N., Myint, M., Yonemori, K., Mukada, N., Kanda, Y., Yanou, A., and Minami, M. (2019). “Dual-Eye Vision-Based Docking Experiment in the Sea for Battery Recharging Application.” SICE Journal of Control, Measurement, and System Integration, 12(2), 47–55.
Martin, E. J., Erwin, B., Katija, K., Phung, A., Gonzalez, E., Von Thun, S., Cullen, H., and Haddock, S. H. “A Virtual Reality Video System for Deep Ocean Remotely Operated Vehicles.” Proc., OCEANS 2021: San Diego–Porto, IEEE, 1–6.
Müller, M. (2007). “Dynamic time warping.” Information retrieval for music and motion, 69–84.
Pedersen, S., Liniger, J., Sørensen, F. F., Schmidt, K., von Benzon, M., and Klemmensen, S. S. (2019). “Stabilization of a rov in three-dimensional space using an underwater acoustic positioning system.” IFAC-PapersOnLine, 52(17), 117–122.
Shazali, S. M. “Development of handheld haptics device for driving system of unmanned underwater vehicles.” Proc., MATEC Web of Conferences, EDP Sciences, 06033.
Soylu, S., Proctor, A. A., Podhorodeski, R. P., Bradley, C., and Buckham, B. J. (2016). “Precise trajectory control for an inspection class ROV.” Ocean Engineering, 111, 508–523.
VIVE. (2023). “OpenXR for PC VR.” <https://developer.vive.com/resources/openxr/openxr-pcvr/documentation/>. (January 1st, 2023).
VIVE, H. (2022). “HTC VIVE VR Devices - Premium VR Experience.” <https://www.vive.com/us/?wpsrc=Google%20AdWords&wpcid=16946345271&wpsnetn=g&wpkwn=htc%20vive&wpkmatch=e&wpcrid=594055656895&wpscid=142495177624&wpkwid=kwd-15492340847&gclid=CjwKCAiA76-dBhByEiwAA0_s9V7J5u0MKXvVAFJiaV_8EQ8HT9ZL4ReU_99KFRlqgZWpz9-hjKxcNBoCazwQAvD_BwE>. (December 28th, 2022).
Xia, P., McSweeney, K., Wen, F., Song, Z., Krieg, M., Li, S., Yu, X., Crippen, K., Adams, J., and Du, E. J. “Virtual Telepresence for the Future of ROV Teleoperations: Opportunities and Challenges.” Proc., SNAME 27th Offshore Symposium, OnePetro.
Xia, P., Xu, F., Song, Z., Li, S., and Du, J. (2023). “Sensory augmentation for subsea robot teleoperation.” Computers in Industry, 145, 103836.
Zhou, T., Zhu, Q., and Du, J. (2020). “Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction.” Advanced Engineering Informatics, 46, 101170.
Zhu, Q., Zhou, T., and Du, J. (2022). “Haptics-based force balance controller for tower crane payload sway controls.” Automation in Construction, 144, 104597.
Zhu, Q., Zhou, T., and Du, J. (2022). “Upper-body haptic system for snake robot teleoperation in pipelines.” Advanced Engineering Informatics, 51, 101532.
Information & Authors
Information
Published In
Computing in Civil Engineering 2023
Pages: 488 - 495
History
Published online: Jan 25, 2024
ASCE Technical Topics:
- Business management
- Case studies
- Continuum mechanics
- Control systems
- Design (by type)
- Dynamics (solid mechanics)
- Education
- Engineering fundamentals
- Engineering mechanics
- Highway transportation
- Human and behavioral factors
- Hydraulic engineering
- Hydraulic properties
- Infrastructure
- Load factors
- Methodology (by type)
- Motion (dynamics)
- Practice and Profession
- Research methods (by type)
- Solid mechanics
- Structural design
- Submerging
- Systems engineering
- Systems management
- Training
- Transportation engineering
- Vehicles
- Water and water resources
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.