Attitude Control of a Moving Mass–Actuated UAV Based on Deep Reinforcement Learning
Publication: Journal of Aerospace Engineering
Volume 35, Issue 2
Abstract
A moving mass–actuated unmanned aerial vehicle (MAUAV) is controlled by mass sliders installed inside the airframe and has the advantages of high aerodynamic efficiency and good stealth performance. However, designing a controller for it faces severe challenges due to the strong nonlinearity and coupling of its dynamics. To this end, we proposed an attitude controller based on deep reinforcement learning for the MAUAV. It directly maps the states to the needed deflection of the actuators and is an end-to-end controller. For the sparse reward problem, the reward function required for training is reasonably designed through reward shaping to hasten the algorithm’s training speed. In training, random initialization and parameter perturbation are used to strengthen the final policy’s robustness further. The simulation results tentatively demonstrate that the proposed controller is not only robust but suboptimal. Compared with an active disturbance rejection controller (ADRC) optimized by the particle swarm algorithm, our controller still guarantees a 100% success rate in multiple unlearned scenarios, meaning it has good generalization ability.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This study was supported by the National Natural Science Foundation of China (No. 11572097).
References
Abbeel, P., A. Coates, M. Quigley, and A. Y. Ng. 2006. “An application of reinforcement learning to aerobatic helicopter flight.” In Proc., 19th Int. Conf. on Neural Information Processing Systems, 1–8. Cambridge, MA: MIT Press.
Bou-Ammar, H., H. Voos, and W. Ertel. 2010. “Controller design for quadrotor UAVs using reinforcement learning.” In Proc., 2010 IEEE Int. Conf. on Control Applications, 2130–2135. New York: IEEE.
Calhoun, P., and E. Queen. 2002. “Entry vehicle control system design for the Mars smart lander.” In Proc., AIAA Atmospheric Flight Mechanics Conf. and Exhibit. San Diego: American Institute of Aeronautics and Astronautics.
Chen, L., G. Zhou, X. J. Yan, and D. P. Duan. 2012. “Composite control strategy of stratospheric airships with moving masses.” J. Aircr. 49 (3): 794–801. https://doi.org/10.2514/1.C031364.
Darvishpoor, S., J. Roshanian, and M. Tayefi. 2020. “A novel concept of VTOL bi-rotor UAV based on moving mass control.” Aerosp. Sci. Technol. 107 (Dec): 106238. https://doi.org/10.1016/j.ast.2020.106238.
Dogan, A., and S. A. Erturk. 2014. “Trimming mass-actuated airplane in turns with zero side slip angle.” In Proc., AIAA Atmospheric Flight Mechanics Conf. San Diego: American Institute of Aeronautics and Astronautics.
Erturk, S., O. Daskiran, and A. Dogan. 2012. “Trim analysis of a moving-mass actuated airplane.” In Proc., AIAA Atmospheric Flight Mechanics Conf. San Diego: American Institute of Aeronautics and Astronautics.
Erturk, S. A., and A. Dogan. 2013. “Trim analysis of a moving-mass actuated airplane in steady turn.” In Proc., 51st AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition. San Diego: American Institute of Aeronautics and Astronautics.
Erturk, S. A., and A. Dogan. 2015. “Propeller torque effect on cruise trim of standard and mass-actuated airplane.” In Proc., AIAA Atmospheric Flight Mechanics Conf. San Diego: American Institute of Aeronautics and Astronautics.
Erturk, S. A., and A. Dogan. 2016. “Propeller torque effect on steady-state turn trim of standard and mass-actuated airplane.” In Proc., AIAA Atmospheric Flight Mechanics Conf. San Diego: American Institute of Aeronautics and Astronautics.
Erturk, S. A., and A. Dogan. 2017a. “Trim analyses of mass-actuated airplane in cruise and steady-state turn.” J. Aircr. 54 (4): 1587–1594. https://doi.org/10.2514/1.C034200.
Erturk, S. A., and A. Dogan. 2017b. “Trim analyses of mass-actuated airplane in steady-state climb and descent.” In Proc., AIAA Atmospheric Flight Mechanics Conf. San Diego: American Institute of Aeronautics and Astronautics.
Erturk, S. A., and A. Dogan. 2017c. “Dynamic simulation and control of mass-actuated airplane.” J. Guidance Control Dyn. 40 (8): 1939–1953. https://doi.org/10.2514/1.G002658.
Furfaro, R., A. Scorsoglio, R. Linares, and M. Massari. 2020. “Adaptive generalized ZEM-ZEV feedback guidance for planetary landing via a deep reinforcement learning approach.” Acta Astronaut. 171 (Jun): 156–171. https://doi.org/10.1016/j.actaastro.2020.02.051.
Gaudet, B., R. Linares, and R. Furfaro. 2020a. “Adaptive guidance and integrated navigation with reinforcement meta-learning.” Acta Astronaut. 169 (Apr): 180–190. https://doi.org/10.1016/j.actaastro.2020.01.007.
Gaudet, B., R. Linares, and R. Furfaro. 2020b. “Deep reinforcement learning for six degree-of-freedom planetary landing.” Adv. Space Res. 65 (7): 1723–1741. https://doi.org/10.1016/j.asr.2019.12.030.
Gaudet, B., R. Linares, and R. Furfaro. 2020c. “Terminal adaptive guidance via reinforcement meta-learning: Applications to autonomous asteroid close-proximity operations.” Acta Astronaut. 171 (Jun): 1–13. https://doi.org/10.1016/j.actaastro.2020.02.036.
Gong, L., Q. Wang, C. Hu, and C. Liu. 2020. “Switching control of morphing aircraft based on Q-learning.” Chin. J. Aeronaut. 33 (2): 672–687. https://doi.org/10.1016/j.cja.2019.10.005.
Guo, T., N. Jiang, B. Li, X. Zhu, Y. Wang, and W. Du. 2021. “UAV navigation in high dynamic environments: A deep reinforcement learning approach.” Chin. J. Aeronaut. 34 (2): 479–489. https://doi.org/10.1016/j.cja.2020.05.011.
Haus, T., M. Orsag, and S. Bogdan. 2016a. “Design considerations for a large quadrotor with moving mass control.” In Proc., 2016 Int. Conf. on Unmanned Aircraft Systems, 1327–1334. New York: IEEE.
Haus, T., N. Prkut, K. Borovina, B. Maric, M. Orsag, and S. Bogdan. 2016b. “A novel concept of attitude control for large multirotor-UAVs based on moving mass control.” In Proc., 2016 24th Mediterranean Conf. on Control and Automation, 832–839. New York: IEEE.
Huang, X., J. Liu, C. Jia, Z. Wang, and J. Zhang. 2020. “Deep deterministic policy gradient for UAV control.” [In Chinese.] Acta Aeronaut. Astronaut. Sin. 42 (10): 524688. https://doi.org/10.7527/S1000-6893.2020.24688.
Li, B., and T.-C. Su. 2017. “Heading autopilot of autonomous underwater vehicles with internal moving mass.” J. Comput. Nonlinear Dyn. 12 (2): 021003. https://doi.org/10.1115/1.4034727.
Li, J., C. Gao, T. Feng, and W. Jing. 2018. “Novel moving mass flight vehicle and its equivalent experiment.” J. Dyn. Syst. Meas. Control 140 (11): 111010. https://doi.org/10.1115/1.4040326.
Li, J., C. Gao, W. Jing, and P. Wei. 2017. “Dynamic analysis and control of novel moving mass flight vehicle.” Acta Astronaut. 131 (Feb): 36–44. https://doi.org/10.1016/j.actaastro.2016.11.023.
Li, Q., S. Xie, J. Luo, P. Wu, and J. Chen. 2014. “Pitch reduction system design and control for an underwater vehicle.” In Proc., 2014 IEEE Int. Conf. on Mechatronics and Automation. New York: IEEE.
Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2015. “Continuous control with deep reinforcement learning.” Preprint, submitted September 09, 2015. http://arxiv.org/abs/1509.02971.
Menon, P. K., G. D. Sweriduk, E. J. Ohlmeyer, and D. S. Malyevac. 2004. “Integrated guidance and control of moving-mass actuated kinetic warheads.” J. Guidance Control Dyn. 27 (1): 118–126. https://doi.org/10.2514/1.9336.
Mnih, V., K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013. “Playing atari with deep reinforcement learning.” Preprint, submitted December 19, 2013. http://arxiv.org/abs/1312.5602.
Petsopoulos, T., F. J. Regan, and J. Barlow. 1996. “Moving-mass roll control system for fixed-trim re-entry vehicle.” J. Spacecraft Rockets 33 (1): 54–60. https://doi.org/10.2514/3.55707.
Qiu, X., M. Zhang, W. Jing, and C. Gao. 2021. “Dynamics and adaptive sliding mode control of a mass-actuated fixed-wing UAV.” Int. J. Aeronaut. Space Sci. 22 (Mar): 886–897. https://doi.org/10.1007/s42405-020-00344-w.
Robinett, R. D., B. R. Sturgis, and S. A. Kerr. 1996. “Moving mass trim control for aerospace vehicles.” J. Guidance Control Dyn. 19 (5): 1064–1070. https://doi.org/10.2514/3.21746.
Rogers, J., and M. Costello. 2008. “Control authority of a projectile equipped with a controllable internal translating mass.” J. Guidance Control Dyn. 31 (5): 1323–1333. https://doi.org/10.2514/1.33961.
Schulman, J., F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. “Proximal policy optimization algorithms.” Preprint, submitted July 20, 2017. http://arxiv.org/abs/1707.06347.
Sutton, R. S., and A. G. Barto. 2018. Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
Sutton, R. S., A. G. Barto, and R. J. Williams. 1992. “Reinforcement learning is direct adaptive optimal control.” IEEE Control Syst. Mag. 12 (2): 19–22. https://doi.org/10.1109/37.126844.
Tan, D., P. Yavarow, and A. Erturk. 2017. “Nonlinear structural dynamics of macro-fiber composite cantilevers for resonant actuation.” In Proc., ASME 2017 Conf. on Smart Materials, Adaptive Structures and Intelligent Systems. New York: ASME.
Vengate, S. R., S. A. Erturk, and A. Dogan. 2016. “Development and flight test of moving-mass actuated unmanned aerial vehicle.” In Proc., AIAA Atmospheric Flight Mechanics Conf. San Diego: American Institute of Aeronautics and Astronautics.
Wang, C., L. Wu, C. Yan, Z. Wang, H. Long, and C. Yu. 2020. “Coactive design of explainable agent-based task planning and deep reinforcement learning for human-UAVs teamwork.” Chin. J. Aeronaut. 33 (11): 2930–2945. https://doi.org/10.1016/j.cja.2020.05.001.
Wang, X., W. Zhou, R. Mu, and Z. Wu. 2020a. “Modeling and simulation of mass-actuated flexible aircraft for roll control.” Aerosp. Sci. Technol. 107 (Dec): 106254. https://doi.org/10.1016/j.ast.2020.106254.
Wang, X., W. Zhou, R. Mu, and Z. Wu. 2020b. “A new deformation control approach for flexible wings using moving masses.” Aerosp. Sci. Technol. 106 (Nov): 106118. https://doi.org/10.1016/j.ast.2020.106118.
Xiang, X., C. Yan, C. Wang, and D. Yi. 2021. “Towards coordination control for fixed-wing UAV formation through deep reinforcement learning.” [In Chinese.] Acta Aeronaut. Astronaut. Sin. 42 (4): 524009. https://doi.org/10.7527/S1000-6893.2020.24009.
Information & Authors
Information
Published In
Journal of Aerospace Engineering
Volume 35 • Issue 2 • March 2022
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Jan 19, 2021
Accepted: Sep 27, 2021
Published online: Dec 1, 2021
Published in print: Mar 1, 2022
Discussion open until: May 1, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Jin Tang, Nianhao Xie, Kebo Li, Yangang Liang, Xinjie Shen, Trajectory Tracking Control for Fixed-Wing UAV Based on DDPG, Journal of Aerospace Engineering, 10.1061/JAEEEZ.ASENG-5286, 37, 3, (2024).
- Cheng Li, Ren Yu, Wenmin Yu, Tianshu Wang, Pressure control of Once-through steam generator using Proximal policy optimization algorithm, Annals of Nuclear Energy, 10.1016/j.anucene.2022.109232, 175, (109232), (2022).