Human Autonomy Teaming for ROV Shared Control
Publication: Journal of Computing in Civil Engineering
Volume 38, Issue 4
Abstract
Remotely operated vehicle (ROV) is a widely used subsea vehicle in offshore oil and gas industries to assist in the development and inspection of offshore oil fields, given its agility, safety, and endurance. As the offshore industry and subsea engineering fields advance, the demands on ROV technology, particularly in terms of control accuracy and safety, have been greatly increasing. Traditional methods require ROV operators to be stationed on vessels in challenging conditions, relying on video streams that offer limited spatial information about the ROV working environment. All these factors make conventional joystick controls even harder for novices. This research aims to propose a novel approach to human-autonomy collaboration aiming to diminish the learning and operational burdens on operators. A virtual reality (VR)-based sensory augmentation and body motion teleoperation method was applied, which allows operators to teleoperate the ROV in a much more comfortable environment. An interactive user interface was developed to enable seamless engagement with the autonomous system and facilitate dynamic switching between human and autonomous controls. To assess the system, a human subject experiment was conducted to compare the performance of an inspection task between human control, full autonomy, and human autonomy teaming methods. The result indicated that our solution could enhance human understanding of the ROV work status as well as reduce human workload during operation.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or code generated or used during the study are available from the corresponding author by request.
Acknowledgments
This material is supported by the National Science Foundation (NSF) under Grant No. 2128895. Any opinions, findings, conclusions, or recommendations expressed in this article are those of the authors and do not reflect the views of the NSF.
References
Abbass, H. A. 2019. “Social integration of artificial intelligence: Functions, automation allocation logic and human-autonomy trust.” Cognit. Comput. 11 (2): 159–171. https://doi.org/10.1007/s12559-018-9619-0.
Amemiya, T., and T. Maeda. 2009. “Directional force sensation by asymmetric oscillation from a double-layer slider-crank mechanism.” J. Comput. Inf. Sci. Eng. 9 (1): 1–8. https://doi.org/10.1115/1.3072900.
Amundsen, H. B., W. Caharija, and K. Y. Pettersen. 2021. “Autonomous ROV inspections of aquaculture net pens using DVL.” IEEE J. Ocean Eng. 47 (1): 1–19. https://doi.org/10.1109/JOE.2021.3105285.
Azis, F., M. Aras, M. Rashid, M. Othman, and S. Abdullah. 2012. “Problem identification for underwater remotely operated vehicle (ROV): A case study.” Procedia Eng. 41 (Jan): 554–560. https://doi.org/10.1016/j.proeng.2012.07.211.
bHaptics. 2022. “TactSuit X40.” Accessed July 8, 2022. https://www.bhaptics.com/tactsuit/tactsuit-x40.
Bischoff, M. 2021. “ros-sharp.” Accessed May 9, 2023. https://github.com/siemens/ros-sharp.
BlueRobotics. 2021. “BlueROV2 operation.” Accessed December 7, 2021. https://bluerobotics.com/learn/bluerov2-operation/.
Brooks, F. P. 1999. “What’s real about virtual reality?” IEEE Comput. Graph. Appl. 19 (6): 16–27. https://doi.org/10.1109/38.799723.
Brun, L. 2012. “ROV/AUV trends: Market and technology.” Mar. Technol. Rep. 5 (7): 48–51. https://doi.org/10.13140/RG.2.1.4062.5686.
Calhoun, G. L., H. A. Ruff, K. J. Behymer, and E. M. Frost. 2018. “Human-autonomy teaming interface design considerations for multi-unmanned vehicle control.” Theor. Issues Ergon. Sci. 19 (3): 321–352. https://doi.org/10.1080/1463922X.2017.1315751.
Casey, J. 2020. “Drawing the line: Could the subsea industry turn away from oil and gas.” Accessed May 4, 2023. https://www.offshore-technology.com/features/drawing-the-line-could-the-subsea-industry-turn-away-from-oil-and-gas/.
Chellali, R., and K. Baizid. 2011. “What maps and what displays for remote situation awareness and ROV localization?” In Proc., Human Interface and the Management of Information, Interacting with Information: Symp. on Human Interface 2011, Held as Part of HCI International 2011, 364–372. Berlin: Springer.
Chen, J. Y., M. J. Barnes, A. R. Selkowitz, K. Stowers, S. G. Lakhmani, and N. Kasdaglis. 2016. “Human-autonomy teaming and agent transparency.” In Proc., Companion Publication of the 21st Int. Conf. on Intelligent User Interfaces, 28–31. New York: Association for Computing Machinery.
Ciriello, V. M., R. V. Maikala, and N. V. O’Brien. 2013. “Maximal acceptable torques of six highly repetitive hand-wrist motions for male industrial workers.” Hum. Factors 55 (2): 309–322. https://doi.org/10.1177/0018720812454539.
Concannon, D., R. Flynn, and N. Murray. 2019. “A quality of experience evaluation system and research challenges for networked virtual reality-based teleoperation applications.” In Proc., 11th ACM Workshop on Immersive Mixed and Virtual Environment Systems, 10–12. New York: Association for Computing Machinery.
Cuzick, J. 1985. “A Wilcoxon-type test for trend.” Stat. Med. 4 (1): 87–90. https://doi.org/10.1002/sim.4780040112.
Demir, M., N. J. Cooke, and P. G. Amazeen. 2018. “A conceptual model of team dynamical behaviors and performance in human-autonomy teaming.” Cognit. Syst. Res. 52 (Dec): 497–507. https://doi.org/10.1016/j.cogsys.2018.07.029.
Elor, A., T. Thang, B. P. Hughes, A. Crosby, A. Phung, E. Gonzalez, K. Katija, S. H. Haddock, E. J. Martin, and B. E. Erwin. 2021. “Catching Jellies in immersive virtual reality: A comparative teleoperation study of ROVs in underwater capture tasks.” In Proc., 27th ACM Symp. on Virtual Reality Software and Technology, 1–10. New York: Association for Computing Machinery.
Felzmann, H., E. Fosch-Villaronga, C. Lutz, and A. Tamo-Larrieux. 2019. “Robots and transparency: The multiple dimensions of transparency in the context of robot technologies.” IEEE Robot. Autom. Manage. 26 (2): 71–78. https://doi.org/10.1109/MRA.2019.2904644.
Finney, G. R. 2015. “Perceptual-motor dysfunction.” Continuum 21 (3): 678–689. https://doi.org/10.1212/01.CON.0000466660.82284.69.
Guerrero, M., N. Rüther, and R. Szupiany. 2012. “Laboratory validation of acoustic Doppler current profiler (ADCP) techniques for suspended sediment investigations.” Flow Meas. Instrum. 23 (1): 40–48. https://doi.org/10.1016/j.flowmeasinst.2011.10.003.
Harmonic, W. 2022. “Crest ocean system HDRP.” Accessed July 5, 2022. https://assetstore.unity.com/packages/tools/particles-effects/crest-ocean-system-hdrp-164158#description.
Hart, S. G. 2006. “NASA-task load index (NASA-TLX); 20 years later.” In Proc., Human Factors and Ergonomics Society Annual Meeting, 904–908. Los Angeles: SAGE.
Institute. 2018. “ROV pilots and technicians.” Accessed May 5, 2023. https://www.idicharleston.edu/rov-pilots-technicians/.
Khadhraoui, A., L. Beji, S. Otmane, and A. Abichou. 2016. “Stabilizing control and human scale simulation of a submarine ROV navigation.” Ocean Eng. 114 (Mar): 66–78. https://doi.org/10.1016/j.oceaneng.2015.12.054.
Kirsch, W., and W. Kunde. 2013. “Moving further moves things further away in visual perception: Position-based movement planning affects distance judgments.” Exp. Brain Res. 226 (3): 431–440. https://doi.org/10.1007/s00221-013-3455-y.
Lachaud, E., Y. Monbeig, P. Nolleau, A. Hardy, M. Thompson, and M. Lardeux. 2018. “Opportunities and challenges of remote operating a ROV embarked on a USV.” In Proc., Offshore Technology Conf. Richardson, TX: OnePetro, Society of Petroleum Engineers.
Li, B., B. Moridian, and N. Mahmoudian. 2018. “Autonomous oil spill detection: Mission planning for ASVs and AUVs with static recharging.” In Proc., OCEANS 2018 MTS/IEEE Charleston, 1–5. New York: IEEE.
Long, C., M. Hu, X. Qin, and Y. Bian. 2022. “Hierarchical trajectory tracking control for ROVs subject to disturbances and parametric uncertainties.” Ocean Eng. 266 (Dec): 112733. https://doi.org/10.1016/j.oceaneng.2022.112733.
Long, C., X. Qin, Y. Bian, and M. Hu. 2021. “Trajectory tracking control of ROVs considering external disturbances and measurement noises using ESKF-based MPC.” Ocean Eng. 241 (Dec): 109991. https://doi.org/10.1016/j.oceaneng.2021.109991.
Lwin, K. N., M. Myint, K. Yonemori, N. Mukada, Y. Kanda, A. Yanou, and M. Minami. 2019. “Dual-eye vision-based docking experiment in the sea for battery recharging application.” SICE J. Control Meas. Syst. Integr. 12 (2): 47–55. https://doi.org/10.9746/jcmsi.12.47.
Lyons, J. B., K. Sycara, M. Lewis, and A. Capiola. 2021. “Human–autonomy teaming: Definitions, debates, and directions.” Front. Psychol. 12 (May): 589585. https://doi.org/10.3389/fpsyg.2021.589585.
Mack, E. 2015. “Improved GPS could untether VR and revolutionize geolocation.” Accessed August 25, 2023. https://newatlas.com/super-accurate-gps/37422/.
Mader, D., R. Blaskow, P. Westfeld, and C. Weller. 2016. “Potential of UAV-based laser scanner and multispectral camera data in building inspection.” In International archives of the photogrammetry, remote sensing and spatial information sciences, 41. Amsterdam, Netherlands: International Society for Photogrammetry and Remote Sensing.
Marini, F., and B. Walczak. 2015. “Particle swarm optimization (PSO): A tutorial.” Chemometr. Intell. Lab. Syst. 149 (Dec): 153–165. https://doi.org/10.1016/j.chemolab.2015.08.020.
Martin, E. J., B. Erwin, K. Katija, A. Phung, E. Gonzalez, S. Von Thun, H. Cullen, and S. H. Haddock. 2021. “A virtual reality video system for deep ocean remotely operated vehicles.” In Proc., OCEANS 2021: San Diego–Porto, 1–6. New York: IEEE.
Masnadi, S., K. Pfeil, J.-V. T. Sera-Josef, and J. LaViola. 2022. “Effects of field of view on egocentric distance perception in virtual reality.” In Proc., 2022 CHI Conf. on Human Factors in Computing Systems, 1–10. New York: Association for Computing Machinery.
McGee, M. G. 1979. “Human spatial abilities: Psychometric studies and environmental, genetic, hormonal, and neurological influences.” Psychol. Bull. 86 (5): 889. https://doi.org/10.1037/0033-2909.86.5.889.
McNeese, N. J., M. Demir, E. K. Chiou, and N. J. Cooke. 2021. “Trust and team performance in human–autonomy teaming.” Int. J. Electron. Commerce 25 (1): 51–72. https://doi.org/10.1080/10864415.2021.1846854.
Meireles, M., R. Lourenço, A. Dias, J. M. Almeida, H. Silva, and A. Martins. 2014. “Real time visual SLAM for underwater robotic inspection.” In Proc., 2014 Oceans-St. John’s, 1–5. New York: IEEE.
Müller, M. 2007. “Dynamic time warping.” In Information retrieval for music and motion, 69–84. Berlin: Springer.
Nitonye, S., S. Adumene, C. U. Orji, and A. Effiong Udo. 2021. “Operational failure assessment of remotely operated vehicle (ROV) in harsh offshore environments.” Pomorstvo 35 (2): 275–286. https://doi.org/10.31217/p.35.2.10.
NOAA (National Oceanic and Atmospheric Administration). 2021. “What is an ROV?” Accessed December 3, 2021. https://oceanexplorer.noaa.gov/facts/rov.html.
Oceaneering. 2022. ROV personnel and training. Accessed October 26, 2022. https://www.oceaneering.com/rov-services/rov-personnel-training/.
O’Neill, T., N. McNeese, A. Barron, and B. Schelble. 2022. “Human–autonomy teaming: A review and analysis of the empirical literature.” Hum. Factors 64 (5): 904–938. https://doi.org/10.1177/0018720820960865.
Patiris, I. 2015. “ROV, remote operated vehicle.” Accessed May 5, 2023. https://www.theseus.fi/bitstream/handle/10024/86911/Ioannis_Patiris.pdf?sequence=1.
Rahman, S. M., Z. Liao, L. Jiang, and Y. Wang. 2016. “A regret-based autonomy allocation scheme for human-robot shared vision systems in collaborative assembly in manufacturing.” In Proc., 2016 IEEE Int. Conf. on Automation Science and Engineering (CASE), 897–902. New York: IEEE.
Rakha, T., and A. Gorodetsky. 2018. “Review of unmanned aerial system (UAS) applications in the built environment: Towards automated building inspection procedures using drones.” Autom. Constr. 93 (Jun): 252–264. https://doi.org/10.1016/j.autcon.2018.05.002.
Richman, J. S., D. E. Lake, and J. R. Moorman. 2004. “Sample entropy.” In Methods in enzymology, 172–184. New York: Elsevier.
Roth, E. M., C. Sushereba, L. G. Militello, J. Diiulio, and K. Ernst. 2019. “Function allocation considerations in the era of human autonomy teaming.” J. Cognit. Eng. Decis. Making 13 (4): 199–220. https://doi.org/10.1177/1555343419878038.
Sapp, C. 2023. “Enhanced ROV performance using AR/VR HUDs.” In Proc., Offshore Technology Conf. Richardson, TX: OnePetro, Society of Petroleum Engineers.
Schaefer, K. E., S. G. Hill, and F. G. Jentsch. 2019. “Trust in human-autonomy teaming: A review of trust research from the US Army research laboratory robotics collaborative technology alliance.” In Proc., Advances in Human Factors in Robots and Unmanned Systems: Proc. of the AHFE 2018 Int. Conf. on Human Factors in Robots and Unmanned Systems, July 21-25, 2018, Loews Sapphire Falls Resort at Universal Studios, Orlando, Florida, USA 9, 102–114. Berlin: Springer.
Schjølberg, I., and I. B. Utne. 2015. “Towards autonomy in ROV operations.” IFAC-PapersOnLine 48 (2): 183–188. https://doi.org/10.1016/j.ifacol.2015.06.030.
Shadlen, M. N., and W. T. Newsome. 1996. “Motion perception: Seeing and deciding.” Proc. Natl. Acad. Sci. U.S.A. 93 (2): 628–633. https://doi.org/10.1073/pnas.93.2.628.
Shazali, S. M. 2018. “Development of handheld haptics device for driving system of unmanned underwater vehicles.” In Proc., MATEC Web of Conf., 06033. London: EDP Sciences.
Simpson, T. 2021. “Real-time drone surveillance system for violent crowd behavior unmanned aircraft system (UAS)–human autonomy teaming (HAT).” In Proc., 2021 IEEE/AIAA 40th Digital Avionics Systems Conf. (DASC), 1–9. New York: IEEE.
Skeete, J.-P. 2018. “Level 5 autonomy: The new face of disruption in road transport.” Technol. Forecasting Social Change 134 (Sep): 22–34. https://doi.org/10.1016/j.techfore.2018.05.003.
Song, Z., A. Marburg, and D. Manalang. 2020. “Resident subsea robotic systems: A review.” Mar. Technol. Soc. J. 54 (5): 21–31. https://doi.org/10.4031/MTSJ.54.5.4.
Soylu, S., A. A. Proctor, R. P. Podhorodeski, C. Bradley, and B. J. Buckham. 2016. “Precise trajectory control for an inspection class ROV.” Ocean Eng. 111 (Jan): 508–523. https://doi.org/10.1016/j.oceaneng.2015.08.061.
Tran, N.-H., M.-C. Le, T.-P. Ton, and T.-P. Tran. 2020. “ROV stabilization using an adaptive nonlinear feedback controller.” In Proc., Int. Conf. on Green Technology and Sustainable Development, 144–155. Berlin: Springer.
Trslic, P., M. Rossi, L. Robinson, C. W. O’Donnel, A. Weir, J. Coleman, J. Riordan, E. Omerdic, G. Dooly, and D. Toal. 2020. “Vision based autonomous docking for work class ROVs.” Ocean Eng. 196 (Aug): 106840. https://doi.org/10.1016/j.oceaneng.2019.106840.
Vargas, E., R. Scona, J. S. Willners, T. Luczynski, Y. Cao, S. Wang, and Y. R. Petillot. 2021. “Robust underwater visual SLAM fusing acoustic sensing.” In Proc., 2021 IEEE Int. Conf. on Robotics and Automation (ICRA), 2140–2146. New York: IEEE.
VIVE. 2022. “HTC VIVE VR devices—Premium VR experience.” Accessed December 28, 2022. https://www.vive.com/us/?wpsrc=Google%20AdWords&wpcid=16946345271&wpsnetn=g&wpkwn=htc%20vive&wpkmatch=e&wpcrid=594055656895&wpscid=142495177624&wpkwid=kwd-15492340847&gclid=CjwKCAiA76-dBhByEiwAA0_s9V7J5u0MKXvVAFJiaV_8EQ8HT9ZL4ReU_99KFRlqgZWpz9-hjKxcNBoCazwQAvD_BwE.
Walker, K. L., A. A. Stokes, A. Kiprakis, and F. Giorgio-Serchi. 2020. “Impact of thruster dynamics on the feasibility of ROV station keeping in waves.” In Proc., Global Oceans 2020: Singapore–US Gulf Coast, 1–7. New York: IEEE.
WBOC (World’s Best On the Coast). 2021. “Underwater ROV market size is expected to grow at a CAGR of 8.5% during 2021-2026 with top countries data.” Accessed December 3, 2021. https://www.wboc.com/story/44099670/underwater-rov-market-size-is-expected-to-grow-at-a-cagr-of-85-during-2021-2026-with-top-countries-data.
Williams, T., D. Szafir, T. Chakraborti, and E. Phillips. 2019. “Virtual, augmented, and mixed reality for human-robot interaction (VAM-HRI).” In Proc., 2019 14th ACM/IEEE Int. Conf. on Human-Robot Interaction (HRI), 671–672. New York: IEEE.
Wood, G., S. J. Vine, and M. R. Wilson. 2013. “The impact of visual illusions on perception, action planning, and motor performance.” Attention Percep. Psychophys. 75 (5): 830–834. https://doi.org/10.3758/s13414-013-0489-y.
Xia, P., K. McSweeney, F. Wen, Z. Song, M. Krieg, S. Li, X. Yu, K. Crippen, J. Adams, and E. J. Du. 2022a. “Virtual telepresence for the future of ROV teleoperations: Opportunities and challenges.” In Proc., SNAME 27th Offshore Symp., 1–12. Richardson, TX: OnePetro, Society of Petroleum Engineers.
Xia, P., F. Xu, Z. Song, S. Li, and J. Du. 2023a. “Sensory augmentation for subsea robot teleoperation.” Comput. Ind. 145 (Feb): 103836. https://doi.org/10.1016/j.compind.2022.103836.
Xia, P., F. Xu, T. Zhou, and J. Du. 2022b. “Benchmarking human versus robot performance in emergency structural inspection.” J. Constr. Eng. Manage. 148 (8): 04022070. https://doi.org/10.1061/(ASCE)CO.1943-7862.0002322.
Xia, P., H. You, and J. Du. 2023b. “Visual-haptic feedback for ROV subsea navigation control.” Autom. Constr. 154 (Oct): 104987. https://doi.org/10.1016/j.autcon.2023.104987.
Xia, P., H. You, Y. Ye, and J. Du. 2023c. “ROV teleoperation via human body motion mapping: Design and experiment.” Comput. Ind. 150 (Sep): 103959. https://doi.org/10.1016/j.compind.2023.103959.
Yang, X., I. B. Utne, S. S. Sandøy, M. A. Ramos, and B. Rokseth. 2020. “A systems-theoretic approach to hazard identification of marine systems with dynamic autonomy.” Ocean Eng. 217 (Dec): 107930. https://doi.org/10.1016/j.oceaneng.2020.107930.
Ye, Y., Y. Shi, Y. Lee, G. Burks, D. Srinivasan, and J. Du. 2022. “Exoskeleton training through haptic sensation transfer in immersive virtual environment.” In Proc., Construction Research Congress 2022, 560–569. Reston, VA: ASCE.
Zaman, M. Q., and R. Mardiyanto. 2021. “Development of marker detection method for estimating angle and distance of underwater remotely operated vehicle to buoyant boat.” Int. J. Adv. Intell. Inf. 7 (3): 249–267. https://doi.org/10.26555/ijain.v7i3.455.
Zheng, J., K. Chan, and I. Gibson. 1998. “Virtual reality.” IEEE Potentials 17 (2): 20–23. https://doi.org/10.1109/45.666641.
Zhou, T., P. Xia, Y. Ye, and E. Du. 2023a. “Embodied robot teleoperation based on high-fidelity visual-haptic simulator: Pipe fitting example.” J. Constr. Eng. Manage. 149 (12): 04023129. https://doi.org/10.1061/JCEMD4.COENG-13916.
Zhou, T., P. Xia, Q. Zhu, and J. Du. 2023b. “Cognition-driven navigation assistive system for emergency indoor wayfinding (CogDNA): Proof of concept and evidence.” Saf. Sci. 162 (Jun): 106100. https://doi.org/10.1016/j.ssci.2023.106100.
Zhou, T., Q. Zhu, and J. Du. 2020. “Intuitive robot teleoperation for civil engineering operations with virtual reality and deep learning scene reconstruction.” Adv. Eng. Inf. 46 (Oct): 101170. https://doi.org/10.1016/j.aei.2020.101170.
Zhu, Q., T. Zhou, and J. Du. 2022a. “Haptics-based force balance controller for tower crane payload sway controls.” Autom. Constr. 144 (Dec): 104597. https://doi.org/10.1016/j.autcon.2022.104597.
Zhu, Q., T. Zhou, and J. Du. 2022b. “Upper-body haptic system for snake robot teleoperation in pipelines.” Adv. Eng. Inf. 51 (Jun): 101532. https://doi.org/10.1016/j.aei.2022.101532.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 38 • Issue 4 • July 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Sep 30, 2023
Accepted: Jan 10, 2024
Published online: Mar 27, 2024
Published in print: Jul 1, 2024
Discussion open until: Aug 27, 2024
ASCE Technical Topics:
- Business management
- Coasts, oceans, ports, and waterways engineering
- Construction engineering
- Construction management
- Highway transportation
- Human and behavioral factors
- Hydraulic engineering
- Hydraulic properties
- Infrastructure
- Inspection
- Occupational safety
- Ocean engineering
- Offshore platforms
- Offshore structures
- Practice and Profession
- Public administration
- Public health and safety
- Safety
- Submerging
- Transportation engineering
- Vehicles
- Water and water resources
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.