An Integrated Tracking Control Approach Based on Reinforcement Learning for a Continuum Robot in Space Capture Missions
Publication: Journal of Aerospace Engineering
Volume 35, Issue 5
Abstract
In this paper, an integrated tracking control approach was developed for a continuum robot in space capture missions. For the configuration of a three-module cable-driven continuum robot, the nonlinear dynamics equations were derived. The uncertain movement of noncooperative debris requires a real-time trajectory planning solution. Therefore, an adaptive controller based on deep reinforcement learning (DRL) is proposed to generate a dynamic controller in continuous action space, where the trajectory planning function is simultaneously integrated into the dynamic solution. To obtain an efficient policy network for the highly nonlinear dynamics model, the rolling optimization method was combined in the DRL method of the deep deterministic policy gradient (DDPG). The DRL controller generated an appropriate control sequence according to the long-term control performance of the robot system and then executed optimal control input according to the rolling optimization. The simulation result shows that the proposed policy network of the improved DDPG controller can reasonably provide the tracking control solution in the noncooperative debris capture mission.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, and code generated or used during the study appear in the published article.
Acknowledgments
This work was supported by the National Natural Science Foundation Key Foundation No. 91748203 and National Outstanding Youth Science Fund Project of the National Natural Science Foundation of China No. 11922203.
References
Böttcher, G., S. Lilge, and J. Burgner-Kahrs. 2021. “Design of a reconfigurable parallel continuum robot with tendon-actuated kinematic chains.” IEEE Rob. Autom. Lett. 6 (2): 1272–1279. https://doi.org/10.1109/LRA.2021.3057557.
Burgner-Kahrs, J., D. C. Rucker, and H. Choset. 2015. “Continuum robots for medical applications: A survey.” IEEE Trans. Rob. 31 (6): 1261–1280. https://doi.org/10.1109/TRO.2015.2489500.
Chu, M., and X. Wu. 2018. “Modeling and self-learning soft-grasp control for free-floating space manipulator during target capturing using variable stiffness method.” IEEE Access 6: 7044–7054. https://doi.org/10.1109/ACCESS.2017.2788400.
Eren, U., A. Prach, B. B. Koçer, S. V. Raković, E. Kayacan, and B. Açıkmeşe. 2017. “Model predictive control in aerospace systems: Current state and opportunities.” J. Guid. Control Dyn. 40 (7): 1541–1566. https://doi.org/10.2514/1.G002507.
Fabio, M., T. Felix, G. Michael, and P. Jan. 2018. “Domain randomization for simulation-based policy optimization with transferability assessment.” In Proc., Conf. on Robot Learning, 700–713. Seattle: Amazon.
Fazel, M., R. Ge, S. M. Kakade, and M. Mesbahi. 2018. “Global convergence of policy gradient methods for the linear quadratic regulator.” Preprint, submitted January 15, 2018. http://arxiv.org/abs/1801.05039.
Flores-Abad, A., M. A. García Terán, I. U. Ponce, and M. Nandayapa. 2021. “Compliant force sensor-less capture of an object in orbit.” IEEE Trans. Aerosp. Electron. Syst. 57 (1): 497–505. https://doi.org/10.1109/TAES.2020.3027108.
Greer, J., T. K. Morimoto, A. M. Okamura, and E. W. Hawkes. 2017. “Series pneumatic artificial muscles (sPAMs) and application to a soft continuum robot.” In Proc., 2017 IEEE Int. Conf. on Robotics and Automation (ICRA), 5503–5510. New York: IEEE. https://doi.org/10.1109/ICRA.2017.7989648.
Gu, S., E. Holly, T. Lillicrap, and S. Levine. 2017. “Deep reinforcement learning for robotic manipulation with asynchronous off-policy updates.” In Proc., 2017 IEEE Int. Conf. on Robotics and Automation (ICRA), 3389–3396. New York: IEEE. https://doi.org/10.1109/ICRA.2017.798938.
Hartmann, G., Z. Shiller, and A. Azaria. 2019. “Deep reinforcement learning for time optimal velocity control using prior knowledge.” In Proc., 2019 IEEE 31st Int. Conf. on Tools with Artificial Intelligence (ICTAI), 186–193. New York: IEEE. https://doi.org/10.1109/ICTAI.2019.00034.
Heidari, H., M. Pouria, S. Sharifi, and M. Karami. 2018. “Design and fabrication of robotic gripper for grasping in minimizing contact force.” Adv. Space Res. 61 (5): 1359–1370. https://doi.org/10.1016/j.asr.2017.12.024.
Huang, P., D. Wang, Z. Meng, F. Zhang, and Z. Liu. 2016. “Impact dynamic modeling and adaptive target capturing control for tethered space robots with uncertainties.” IEEE/ASME Trans. Mechatron. 21 (5): 2260–2271. https://doi.org/10.1109/TMECH.2016.2569466.
Jiang, D., Z. Cai, H. Peng, and Z. Wu. 2021. “Coordinated control based on reinforcement learning for dual-arm continuum manipulators in space capture missions.” J. Aerosp. Eng. 34 (6): 04021087. https://doi.org/10.1061/(ASCE)AS.1943-5525.0001335.
Kahn, G., A. Villaflor, B. Ding, P. Abbeel, and S. Levine. 2018. “Self-supervised deep reinforcement learning with generalized computation graphs for robot navigation.” In Proc., 2018 IEEE Int. Conf. on Robotics and Automation (ICRA), 5129–5136. New York: IEEE. https://doi.org/10.1109/ICRA.2018.8460655.
Kupcsik, A., M. P. Deosemroth, J. Peter, A. P. Loh, and G. Neumann. 2017. “Model-based contextual policy search for data-efficient generalization of robot skills.” Artif. Intell. 247 (Jun): 415–439. https://doi.org/10.1016/j.artint.2014.11.005.
Li, H., W. Yan, and Y. Shi. 2017. “Continuous-time model predictive control of under-actuated spacecraft with bounded control torques.” Automatica 75 (Jan): 144–153. https://doi.org/10.1016/j.automatica.2016.09.024.
Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2016. “Continuous control with deep reinforcement learning.” In Proc., Int. Conf. on Learning Representations. Menlo Park, CA: Facebook.
Mu, Z., T. Liu, W. Xu, Y. Lou, and B. Liang. 2019. “Dynamic feedforward control of spatial cable-driven hyper-redundant manipulators for on-orbit servicing.” Robotica 37 (1): 18–38. https://doi.org/10.1017/S026357471800084X.
Mu, Z., H. Yuan, W. Xu, T. Liu, and B. Liang. 2020. “A segmented geometry method for kinematics and configuration planning of spatial hyper-redundant manipulators.” IEEE Trans. Syst. Man Cybern. Syst. 50 (5): 1746–1756. https://doi.org/10.1109/TSMC.2017.2784828.
Nagabandi, A., G. Kahn, R. S. Fearing, and S. Levine. 2018. “Neural network dynamics for model-based deep reinforcement learning with model-free fine-tuning.” In Proc., 2018 IEEE International Conference on Robotics and Automation (ICRA), 7559–7566. New York: IEEE. https://doi.org/10.1109/ICRA.2018.8463189.
Nishida, S., and K. Satomi. 2011. “Strategy for capturing of a tumbling space debris.” Acta Astronaut. 68 (1–2): 113–120. https://doi.org/10.1016/j.actaastro.2010.06.045.
Peng, H., F. Li, J. Li, and Z. Ju. 2020a. “A symplectic instantaneous optimal control for robot trajectory tracking with differential-algebraic equation models.” IEEE Trans. Ind. Electron. 67 (5): 3819–3829. https://doi.org/10.1109/TIE.2019.2916390.
Peng, J., W. Xu, T. Liu, H. Yuan, and B. Liang. 2021. “End-effector pose and arm-shape synchronous planning methods of a hyper-redundant manipulator for spacecraft repairing.” Mech. Mach. Theory 155 (Jan): 104062. https://doi.org/10.1016/j.mechmachtheory.2020.104062.
Peng, J., W. Xu, T. Yang, Z. Hu, and B. Liang. 2020b. “Dynamic modeling and trajectory tracking control method of segmented linkage cable-driven hyper-redundant robot.” Nonlinear Dyn. 101 (1): 233–253. https://doi.org/10.1007/s11071-020-05764-7.
Robinson, G., and J. Davies. 1999. “Continuum robots: A state of the art.” In Vol. 4 of Proc., 1999 IEEE Int. Conf. on Robotics and Automation, 2849–2854. New York: IEEE. https://doi.org/10.1109/ROBOT.1999.774029.
Rone, W., and P. Ben-Tzvi. 2014. “Continuum robot dynamics utilizing the principle of virtual power.” IEEE Trans. Rob. 30 (1): 275–287. https://doi.org/10.1109/TRO.2013.2281564.
Sangeetha, G., K. Nishank, P. R. Hari, and S. Sasikumar. 2018. “Implementation of a stereo vision based system for visual feedback control of robotic arm for space manipulations.” Procedia Comput. Sci. 133: 1066–1073. https://doi.org/10.1016/j.procs.2018.07.031.
Seleem, I., H. El-Hussieny, S. F. M. Assal, and H. Ishii. 2020. “Development and stability analysis of an imitation learning-based pose planning approach for multi-section continuum robot.” IEEE Access 8: 99366–99379. https://doi.org/10.1109/ACCESS.2020.2997636.
Shah, P., M. Fiser, A. Faust, J. Kew, and D. Hakkani-Tur. 2018. “FollowNet: Robot navigation by following natural language directions with deep reinforcement learning.” Preprint, submitted May 16, 2018. http://arxiv.org/abs/1805.06150.
Shahid, A., L. Roveda, D. Piga, and F. Braghin. 2020. “Learning continuous control actions for robotic grasping with reinforcement learning.” In Proc., 2020 IEEE Int. Conf. on Systems, Man, and Cybernetics (SMC), 4066–4072. New York: IEEE. https://doi.org/10.1109/SMC42975.2020.9282951.
Thuruthel, T., E. Falotico, M. Cianchetti, F. Renda, and C. Laschi. 2016. “Learning global inverse statics solution for a redundant soft robot.” In Vol. 2 of Proc., 13th Int. Conf. on Informatics in Control, Automation and Robotics, 303–310. Berlin: Springer.
Williams, G., A. Aldrich, and E. A. Theodorou. 2017. “Model predictive path integral control: From theory to parallel computation.” J. Guid. Control Dyn. 40 (2): 344–357. https://doi.org/10.2514/1.G001921.
Yip, M., J. Sganga, and D. Camarillo. 2017. “Autonomous control of continuum robot manipulators for complex cardiac ablation tasks.” J. Med. Rob. Res. 2 (1): 1750002. https://doi.org/10.1142/S2424905X17500027.
Zhang, T., G. Kahn, S. Levine, and P. Abbeel. 2016 “Learning deep control policies for autonomous aerial vehicles with MPC-guided policy search.” In Proc., 2016 IEEE Int. Conf. on Robotics and Automation (ICRA), 528–535. New York: IEEE. https://doi.org/10.1109/ICRA.2016.7487175.
Zhou, M., X. Qu, and S. Jin. 2017. “On the impact of cooperative autonomous vehicles in improving freeway merging: A modified intelligent driver model-based approach.” IEEE Trans. Intell. Transp. Syst. 18 (6): 1422–1428. https://doi.org/10.1109/TITS.2016.2606492.
Zhu, Y., D. Zhao, H. He, and J. Ji. 2017. “Event-triggered optimal control for partially unknown constrained-input systems via adaptive dynamic programming.” IEEE Trans. Ind. Electron. 64 (5): 4101–4109. https://doi.org/10.1109/TIE.2016.2597763.
Zhu, Y., D. Zhao, X. Yang, and Q. Zhang. 2018. “Policy iteration for H optimal control of polynomial nonlinear systems via sum of squares programming.” IEEE Trans. Cybern. 48 (2): 500–509. https://doi.org/10.1109/TCYB.2016.2643687.
Information & Authors
Information
Published In
Journal of Aerospace Engineering
Volume 35 • Issue 5 • September 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Sep 25, 2021
Accepted: Jan 18, 2022
Published online: Jun 2, 2022
Published in print: Sep 1, 2022
Discussion open until: Nov 2, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Yifei Li, Erik-Jan van Kampen, Stability Analysis for Incremental Adaptive Dynamic Programming with Approximation Errors, Journal of Aerospace Engineering, 10.1061/JAEEEZ.ASENG-5097, 37, 1, (2024).