TD3-Based Model Predictive Control for Satellite Formation-Keeping
Publication: Journal of Aerospace Engineering
Volume 37, Issue 6
Abstract
The escalating prevalence of formation flights in space missions has led researchers to intensify their focus on designing optimal control systems for satellite formation motion along reference orbits, with the aim of reducing tracking error and energy consumption. However, conventional controllers typically excel at optimizing only one of these objectives, and the manual parameter tuning of such controllers proves to be a challenging task. In this paper, we introduce a novel approach, the twin delayed deep deterministic policy gradient-based model predictive control (TD3-MPC) method. To tackle the multiobjective formation-keeping challenge, a linear model predictive controller based on the satellite’s dynamics had been developed. Subsequently, a cost function is formulated to facilitate the optimization of multiple objectives, specifically tracking error and fuel consumption. In addressing the intricate issue of controller parameter tuning, we employ reinforcement learning and design a reward function reflective of the TD3 algorithm’s controller performance. Simulation results underscore the superior performance of the proposed TD3-MPC algorithm compared to the linear model predictive controller, achieving a notable 27.83% reduction in tracking error and a substantial 48.30% decrease in fuel consumption under large error condition and 3.67% reduction in tracking error and a substantial 22.27% decrease in fuel consumption under small error condition. By effectively combining the strengths of reinforcement learning and model predictive control, TD3-MPC enhances the satellite’s ability to adhere more precisely to its intended trajectory, thereby ensuring the stability and desired operational performance of the satellite formation.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work is supported by the National Natural Science Foundation of China (No. U22B2013), the China Postdoctoral Science Foundation (No. 2021M692589).
References
Alfriend, K., S. R. Vadali, P. Gurfil, J. How, and L. Breger. 2009. Spacecraft formation flying: Dynamics, control and navigation. Amsterdam, Netherlands: Elsevier.
Alhajeri, M., and M. Soroush. 2020. “Tuning guidelines for model-predictive control.” Ind. Eng. Chem. Res. 59 (10): 4177–4191. https://doi.org/10.1021/acs.iecr.9b05931.
Araguz, C., E. Bou-Balust, and E. Alarcón. 2018. “Applying autonomy to distributed satellite systems: Trends, challenges, and future prospects.” Syst. Eng. 21 (5): 401–416. https://doi.org/10.1002/sys.21428.
Avanzini, G., and M. Fedi. 2013. “Refined dynamical analysis of multi-tethered satellite formations.” Acta Astronaut. 84 (Mar): 36–48. https://doi.org/10.1016/j.actaastro.2012.10.031.
Bagheri, P., and A. Khaki-Sedigh. 2015. “Closed form tuning equations for model predictive control of first-order plus fractional dead time models.” Int. J. Control Autom. Syst. 13 (Feb): 73–80. https://doi.org/10.1007/s12555-014-0007-6.
Boggs, P. T., and J. W. Tolle. 1995. “Sequential quadratic programming.” Acta Numer. 4 (Jan): 1–51. https://doi.org/10.1017/S0962492900002518.
Chai, R., A. Savvaris, A. Tsourdos, and S. Chai. 2017. “Multi-objective trajectory optimization of space manoeuvre vehicle using adaptive differential evolution and modified game theory.” Acta Astronaut. 136 (Jul): 273–280. https://doi.org/10.1016/j.actaastro.2017.02.023.
Chiniforoushan, M., K. Raissi, and M. Mortazavi. 2021. “Fault-tolerant model-free predictive control of spacecraft coupled rotational–translational relative motion.” J. Aerosp. Eng. 34 (6): 04021092. https://doi.org/10.1061/(ASCE)AS.1943-5525.0001327.
Cho, H., and A. Yu. 2009. “New approach to satellite formation-keeping: Exact solution to the full nonlinear problem.” J. Aerosp. Eng. 22 (4): 445–455. https://doi.org/10.1061/(ASCE)AS.1943-5525.0000013.
Chobotov, V. A. 2002. Orbital mechanics. Reston, VA: American Institute of Aeronautics and Astronautics.
Chowdhury, M. A., and Q. Lu. 2022. “A novel entropy-maximizing TD3-based reinforcement learning for automatic PID tuning.” In Proc., 2023 American Control Conf. (ACC), 2763–2768. New York: IEEE.
Elkins, J. G., R. Sood, and C. Rumpf. 2020. “Autonomous spacecraft attitude control using deep reinforcement learning.” In Proc., Int. Astronautical Congress. Cologne, Germany: German Aerospace Center.
Elkins, J. G., R. Sood, and C. Rumpf. 2022. “Bridging reinforcement learning and online learning for spacecraft attitude control.” J. Aerosp. Inf. Syst. 19 (1): 62–69. https://doi.org/10.2514/1.I010958.
Eren, U., A. Prach, B. B. Koçer, S. V. Raković, E. Kayacan, and B. Açıkmeşe. 2017. “Model predictive control in aerospace systems: Current state and opportunities.” J. Guid. Control Dyn. 40 (7): 1541–1566. https://doi.org/10.2514/1.G002507.
Fontes, R. M., M. A. F. Martins, and D. Odloak. 2019. “An automatic tuning method for model predictive control strategies.” Ind. Eng. Chem. Res. 58 (47): 21602–21613. https://doi.org/10.1021/acs.iecr.9b03502.
Fujimoto, S., H. van Hoof, and D. Meger. 2018. “Addressing function approximation error in actor-critic methods.” In Proc., Int. Conf. on Machine Learning, 1587–1596. New York: Proceedings of Machine Learning Research.
He, N., D. Shi, J. Wang, M. Forbes, J. Backström, and T. Chen. 2015. “Automated two-degree-of-freedom model predictive control tuning.” Ind. Eng. Chem. Res. 54 (43): 10811–10824. https://doi.org/10.1021/acs.iecr.5b02569.
Hu, Q., X. Shao, and L. Guo. 2023. Intelligent autonomous control of spacecraft with multiple constraints. Singapore: Springer.
Huang, H., W.-W. Cai, and L.-P. Yang. 2020. “6-DOF formation keeping control for an invariant three-craft triangular electromagnetic formation.” Adv. Space Res. 65 (1): 312–325. https://doi.org/10.1016/j.asr.2019.09.033.
Imoto, Y., S. Satoh, T. Obata, and K. Yamada. 2023. “Optimal constellation design based on satellite ground tracks for Earth observation missions.” Acta Astronaut. 207 (Jun): 1–9. https://doi.org/10.1016/j.actaastro.2023.02.040.
Lee, K., C. Park, S.-Y. Park, and D. J. Scheeres. 2014. “Optimal tracking and formation keeping near a general Keplerian orbit under nonlinear perturbations.” Adv. Space Res. 54 (6): 1019–1028. https://doi.org/10.1016/j.asr.2014.06.010.
Lee, K. W., and S. N. Singh. 2018. “Attractive manifold-based noncertainty-equivalence adaptive spacecraft formation flying using output feedback.” In Proc., 2018 AIAA Guidance, Navigation, and Control Conf., AIAA SciTech Forum. Reston, VA: American Institute of Aeronautics and Astronautics.
Li, H., Q. Zong, and X. Zhang. 2022. “Anti-collision trajectory planning for satellite formation reconstruction based on deep reinforcement learning.” In Proc., 2022 41st Chinese Control Conference (CCC), 4672–4677. New York: IEEE.
Li, P., and Z. H. Zhu. 2018. “Model predictive control for spacecraft rendezvous in elliptical orbit.” Acta Astronaut. 146 (May): 339–348. https://doi.org/10.1016/j.actaastro.2018.03.025.
Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2019. “Continuous control with deep reinforcement learning.” Preprint, submitted September 9, 2015. https://arxiv.org/abs/1509.02971.
Liu, H., Z. Chen, X. Wang, and Z. Sun. 2023. “Optimal formation control for multiple rotation-translation coupled satellites using reinforcement learning.” Acta Astronaut. 204 (Mar): 583–590. https://doi.org/10.1016/j.actaastro.2022.09.049.
Liu, Y., G. Ma, Y. Lyu, and P. Wang. 2022. “Neural network-based reinforcement learning control for combined spacecraft attitude tracking maneuvers.” Neurocomputing 484 (May): 67–78. https://doi.org/10.1016/j.neucom.2021.07.099.
Maurath, P. R., A. J. Laub, D. E. Seborg, and D. A. Mellichamp. 1988. “Predictive controller design by principal components analysis.” Ind. Eng. Chem. Res. 27 (7): 1204–1212. https://doi.org/10.1021/ie00079a020.
Miller, D., J. A. Englander, and R. Linares. 2019. “Interplanetary low-thrust design using proximal policy optimization.” In Proc., 2019 AAS/AIAA Astrodynamics Specialist Conf. Reston, VA: American Institute of Aeronautics and Astronautics.
Mnih, V., K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013. “Playing atari with deep reinforcement learning.” Preprint, submitted December 19, 2013. https://arxiv.org/abs/1312.5602.
Nair, R. R., and L. Behera. 2016. “Robust adaptive gain higher order sliding mode observer based control-constrained nonlinear model predictive control for spacecraft formation flying.” IEEE/CAA J. Autom. Sin. 5 (1): 367–381. https://doi.org/10.1109/JAS.2016.7510253.
Nguyen, T. T., N. D. Nguyen, and S. Nahavandi. 2020. “Deep reinforcement learning for multiagent systems: A review of challenges, solutions, and applications.” IEEE Trans. Cybern. 50 (9): 3826–3839. https://doi.org/10.1109/TCYB.2020.2977374.
Qiang, S., Y. Changqing, Y. U. Haili, and S. Yunlong. 2017. “Shape remain for near-earth radial equal quality collinear three-craft coulomb force formation.” J. Nanjing Univ. Aeronaut. Astronaut 49: 76–81.
Rodrigues, L., A. Potts, O. Vilcanqui, and A. Sguarezi. 2019. “Tuning a model predictive controller for Doubly Fed Induction Generator employing a constrained genetic algorithm.” IET Electr. Power Appl. 13 (6): 812–819. https://doi.org/10.1049/iet-epa.2018.5922.
Schaub, H., and K. T. Alfriend. 2001. “ invariant relative orbits for spacecraft formations.” Celestial Mech. Dyn. Astron. 79 (Feb): 77–95. https://doi.org/10.1023/A:1011161811472.
Schweighart, S. A., and R. J. Sedwick. 2002. “High-fidelity linearized J model for satellite formation flight.” J. Guid. Control Dyn. 25 (6): 1073–1080. https://doi.org/10.2514/2.4986.
Schwenzer, M., M. Ay, T. Bergs, and D. Abel. 2021. “Review on model predictive control: An engineering perspective.” Int. J. Adv. Manuf. Technol. 117 (5): 1327–1349. https://doi.org/10.1007/s00170-021-07682-3.
Shi, Z., F. Zhao, X. Wang, and Z. Jin. 2022. “Satellite attitude tracking control of moving targets combining deep reinforcement learning and predefined-time stability considering energy optimization.” Adv. Space Res. 69 (5): 2182–2196. https://doi.org/10.1016/j.asr.2021.12.014.
Sulaiman, N. A., and R. B. Mustafa. 2016. “Using harmonic mean to solve multi-objective linear programming problems.” Am. J. Oper. Res. 6 (1): 25–30. https://doi.org/10.4236/ajor.2016.61004.
Sun, H., H. Zhao, K. Huang, S. Zhen, and Y.-H. Chen. 2017. “Adaptive robust constraint-following control for satellite formation flying with system uncertainty.” J. Guid. Control Dyn. 40 (6): 1492–1502. https://doi.org/10.2514/1.G002396.
Vaddi, S. S., S. R. Vadali, and K. T. Alfriend. 2003. “Formation flying: Accommodating nonlinearity and eccentricity perturbations.” J. Guid. Control Dyn. 26 (2): 214–223. https://doi.org/10.2514/2.5054.
Wang, P. K. C., and F. Y. Hadaegh. 1999. “Minimum-fuel formation reconfiguration of multiple free-flying spacecraft.” J. Astronaut. Sci. 47 (Mar): 77–102. https://doi.org/10.1007/BF03546211.
Wang, Z., J. Tian, and J. Fu. 2023. “Fuel optimization schemes for formation reconfiguration in satellite formation flying.” Adv. Space Res. 72 (5): 1496–1508. https://doi.org/10.1016/j.asr.2023.04.028.
Wu, B., G. Xu, and X. Cao. 2016. “Relative dynamics and control for satellite formation: Accommodating perturbation.” J. Aerosp. Eng. 29 (4): 04016011. https://doi.org/10.1061/(ASCE)AS.1943-5525.0000600.
Wu, J., C. Wei, H. Zhang, Y. Liu, M. Zhang, and H. Wang. 2023. “Learning-based spacecraft reactive anti-hostile-rendezvous maneuver control in complex space environments.” Adv. Space Res. 72 (10): 4531–4552. https://doi.org/10.1016/j.asr.2023.08.043.
Xu, N., J. Lin, Y. Miao, and E. Q. Wu. 2019. “Fuel-optimal predictive control of satellite formation keeping based on genetic algorithm.” In Proc., 2019 IEEE 28th Int. Symp. on Industrial Electronics (ISIE), 527–532. New York: IEEE.
Xu, Y., N. Fitz-Coy, R. Lind, and A. Tatsch. 2007. “ control for satellites formation flying.” J. Aerosp. Eng. 20 (1): 10–21. https://doi.org/10.1061/(ASCE)0893-1321(2007)20:1(10).
Yasini, T., J. Roshanian, and A. Taghavipour. 2023. “Improving the low orbit satellite tracking ability using nonlinear model predictive controller and Genetic Algorithm.” Adv. Space Res. 71 (6): 2723–2732. https://doi.org/10.1016/j.asr.2022.11.037.
You, S., C. Wan, R. Dai, and J. R. Rea. 2021. “Learning-based onboard guidance for fuel-optimal powered descent.” J. Guid. Control Dyn. 44 (3): 601–613. https://doi.org/10.2514/1.G004928.
Zhang, Y. L., G. Q. Zeng, Z. K. Wang, and J. G. Hao. 2008. Theory and application of distributed satellite. Beijing: Science Press.
Information & Authors
Information
Published In
Journal of Aerospace Engineering
Volume 37 • Issue 6 • November 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Dec 21, 2023
Accepted: May 14, 2024
Published online: Aug 5, 2024
Published in print: Nov 1, 2024
Discussion open until: Jan 5, 2025
ASCE Technical Topics:
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.