Investigation of Deep Reinforcement Learning for Longitudinal-Axis Flight Control
Publication: Journal of Aerospace Engineering
Volume 37, Issue 2
Abstract
Traditional aerodynamic modeling methods have some difficulties in multiparameter unsteady state-space modeling and control law design, which raise significant challenges in flight control. The rapid development of machine learning provides a new idea for unsteady aerodynamic modeling and control law design, which has significant value for theoretical research and engineering application. In this paper, an unsteady aerodynamic model environment was established based on deep neural network (DNN), and a dynamic computational fluid dynamics (CFD) virtual environment was taken as an approximation of the real environment. A deep reinforcement learning (DRL) longitudinal-axis flight control method was studied on the basis of both environments. The deep deterministic policy gradient (DDPG) algorithm was used to implement flight control, and the effects of different constraints in the reward function on the results of DRL were studied at the same time. The results show that the DDPG agent effectively achieves longitudinal-axis flight control in the model environment. The DDPG agent trained in the model environment was also used for longitudinal-axis flight control in the dynamic CFD virtual environment, and this method provides a reference for the coupling of dynamic CFD and DRL. The control results from the model environment and the dynamic CFD virtual environment are compared, and results show that the DDPG agent has good robustness in both environments.
Practical Applications
The results of this study suggest that machine learning and deep reinforcement learning can effectively solve complicated multiconstraint, uncertain, and sophisticated control tasks. In this paper, a DNN-based unsteady aerodynamic modeling method was established based on dynamic computational fluid dynamics calculation and machine learning, and the results show that the DNN-based unsteady aerodynamic model has advantages in dealing with highly nonlinear data under unsteady conditions. In addition, the deep deterministic policy gradient algorithm can be well applied in longitudinal-axis flight control, and the agent can be trained in DNN-based unsteady aerodynamic model environment. The agent was also used for longitudinal-axis flight control in the dynamic computational fluid dynamics virtual environment, and the results show that the agent has good robustness in both environments. This provides the possibility for the agent to achieve longitudinal-axis flight control in a real environment.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
The work was financially supported by the National Numerical Wind Tunnel Project (Grant No. NNW2019ZT7- B31). This research was also supported in part by the Priority Academic Program Development of Jiangsu Higher Education Institutions. The authors would like to express their gratitude to EditSprings (https://www.editsprings.cn) for the expert linguistic services provided.
References
Aggarwal, S. 2002. System identification of X33 using neural network. The 2002 NASA Faculty Fellowship Program Research Reports. Daytona Beach, FL: Embry Riddle Aero Univ.
Blazek, J. 2001. Computational fluid dynamics: Principles and applications. Amsterdam, Netherlands: Elsevier.
Blom, F. J. 2015. “Considerations on the spring analogy.” Int. J. Numer. Methods Fluids 32 (6): 647–668. https://doi.org/10.1002/(SICI)1097-0363(20000330)32:6%3C647::AID-FLD979%3E3.0.CO;2-K.
Brunton, S. L., B. R. Noack, and P. Koumoutsakos. 2020. “Machine learning for fluid mechanics.” Annu. Rev. Fluid Mech. 52 (1): 477–508. https://doi.org/10.1146/annurev-fluid-010719-060214.
Celis, R. D., P. S. López, and L. Cadarso. 2021. “Sensor hybridization using neural networks for rocket terminal guidance.” Aerosp. Sci. Technol. 111 (2021): 106527. https://doi.org/10.1016/j.ast.2021.106527.
Cheng, L., F. H. Jiang, and J. F. Li. 2020. “A review on the applications of deep learning in aircraft dynamics and control.” Mech. Eng. 42 (3): 267–276. https://doi.org/10.6052/1000-0879-20-077.
Dong, Y., Z. Shi, K. Chen, and Z. Yao. 2021. “Self-learned suppression of roll oscillations based on model-free reinforcement learning.” Aerosp. Sci. Technol. 116 (2021): 106850. https://doi.org/10.1016/j.ast.2021.106850.
Du, X., P. He, and J. Martins. 2021. “Rapid airfoil design optimization via neural networks-based parameterization and surrogate modeling.” Aerosp. Sci. Technol. 113 (2021): 106701. https://doi.org/10.1016/j.ast.2021.106701.
Gaudet, B., R. Furfaro, and R. Linares. 2020a. “Reinforcement learning for angle-only intercept guidance of maneuvering targets.” Aerosp. Sci. Technol. 99 (2020): 105746. https://doi.org/10.1016/j.ast.2020.105746.
Gaudet, B., R. Linares, and R. Furfaro. 2020b. “Deep reinforcement learning for six degree-of-freedom planetary landing.” Adv. Space Res. 65 (7): 1723–1741. https://doi.org/10.1016/j.asr.2019.12.030.
Goman, M., and A. Khrabrov. 1994. “State-space representation of aerodynamic characteristics of an aircraft at high angles of attack.” J. Aircr. 31 (5): 1109–1115. https://doi.org/10.2514/3.46618.
Gong, Z., and H. Shen. 2007. “Structure self-adapting ANN method in modeling of unsteady aerodynamics.” [In Chinese.] Flight Dyn. 25 (4): 13–16.
Heim, E. H., E. Viken, J. M. Brandon, and M. A. Croom. 2018. “NASA’s learn-to-fly project overview.” In Proc., 2018 Atmospheric Flight Mechanics Conf. Reston, VA: American Institute of Aeronautics and Astronautics.
Hwangbo, J., I. Sa, R. Siegwart, and M. Hutter. 2017. “Control of a quadrotor with reinforcement learning.” IEEE Rob. Autom. Lett. 2 (4): 2096–2103. https://doi.org/10.1109/LRA.2017.2720851.
Jameson, A. 1991. “Time dependent calculations using multigrid, with applications to unsteady flows past airfoils and wings.” In Proc., 10th Computational Fluid Dynamics Conf. Reston, VA: American Institute of Aeronautics and Astronautics.
Kaelbling, L. P., M. L. Littman, and A. W. Moore. 1996. “Reinforcement learning: A survey.” J. Artif. Intell. Res. 4 (1): 237–285. https://doi.org/10.1613/jair.301.
Kirkpatrick, K., J. May, and J. Valasek. 2013. “Aircraft system identification using artificial neural networks.” In Proc., 51st AIAA Aerospace Sciences Meeting Including the New Horizons Forum and Aerospace Exposition, 0878. Reston, VA: American Institute of Aeronautics and Astronautics.
Kohonen, T. 1988. “An introduction to neural computing.” Neural Networks 1 (1): 3–16. https://doi.org/10.1016/0893-6080(88)90020-2.
Lazzara, M., M. Chevalier, M. Colombo, J. G. Garcia, C. Laperyre, and O. Teste. 2022. “Surrogate modelling for an aircraft dynamic landing loads simulation using an LSTM AutoEncoder-based dimensionality reduction approach.” Aerosp. Sci. Technol. 126 (2022): 107629. https://doi.org/10.1016/j.ast.2022.107629.
Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2016. “Continuous control with deep reinforcement learning.” In Proc., 2016 Int. Conf. on Learning Representations. Vancouver, BC, Canada: International Machine Learning Society.
Lin, G., and C. E. Lan. 2006. “A generalized dynamic aerodynamic coefficient model for flight dynamics applications.” In Proc., AIAA Atmospheric Flight Mechanics Conf. Reston, VA: American Institute of Aeronautics and Astronautics.
Liu, Z., R. Han, M. Zhang, Y. Zhang, H. Zhou, G. Wang, and G. Chen. 2022. “An enhanced hybrid deep neural network reduced-order model for transonic buffet flow prediction.” Aerosp. Sci. Technol. 126 (2022): 107636. https://doi.org/10.1016/j.ast.2022.107636.
Mnih, V., K. Kavukcuoglu, D. Silver, A. Graves, I. Antonoglou, D. Wierstra, and M. Riedmiller. 2013. “Playing Atari with deep reinforcement learning.” Submitted December 19, 2013. https://doi.org/10.48550/arXiv.1312.5602.
Morelli, E. A. 2018. “Practical aspects of real-time modeling for the learn-to-fly concept.” In Proc., 2018 Atmospheric Flight Mechanics Conf. Reston, VA: American Institute of Aeronautics and Astronautics.
Mysore, S., B. Mabsout, R. Mancuso, and K. Saenko. 2021. “Regularizing action policies for smooth control with reinforcement learning.” In Proc., 2021 Int. Conf. on Robotics and Automation (ICRA). Reston, VA: American Institute of Aeronautics and Astronautics.
Riddick, S. E., R. C. Busan, D. E. Cox, and S. A. Laughter. 2018. “Learn to fly test setup and concept of operations.” In Proc., 2018 Atmospheric Flight Mechanics Conf. Reston, VA: American Institute of Aeronautics and Astronautics.
Shen, S., and J. Xu. 2021. “Adaptive neural network-based active disturbance rejection flight control of an unmanned helicopter.” Aerosp. Sci. Technol. 119 (2021): 107062. https://doi.org/10.1016/j.ast.2021.107062.
Shi, Z., Z. Wang, and J. Li. 2012. “The research of RBFNN in modeling of nonlinear unsteady aerodynamics.” Acta Aerodynamica Sin. 30 (1): 108–112.
Song, X., Y. Wang, Z. Cai, J. Zhao, X. Chen, and D. Dong. 2020. “Landing trajectory tracking control of unmanned aerial vehicle by deep reinforcement learning.” Aeronaut. Sci. Technol. 31 (1): 68–75.
Sørensen, K. A., O. Hassan, K. Morgan, and N. P. Weatherill. 2003. “A multigrid accelerated hybrid unstructured mesh method for 3D compressible turbulent flow.” Comput. Mech. 31 (1–2): 101–114. https://doi.org/10.1007/s00466-002-0397-9.
Stastny, T. J., R. Lykins, and S. Keshmiri. 2013. “Nonlinear parameter estimation of unmanned aerial vehicle in wind shear using artificial neural networks.” In Proc., AIAA Guidance, Navigation, and Control (GNC) Conf., 4768. Reston, VA: American Institute of Aeronautics and Astronautics.
Sutton, R. S., and A. G. Barto. 2018. Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
Suvarna, S., H. Chung, A. Sinha, and R. Pant. 2022. “Revised semi-empirical aerodynamic estimation for modelling flight dynamics of an airship.” Aerosp. Sci. Technol. 126 (2022): 107642. https://doi.org/10.1016/j.ast.2022.107642.
Tobak, M. 1954. On the use of the indicial function concept in the analysis of unsteady motion of wing and wing-tail combinations. NACA R-1188. Washington, DC: National Advisory Committee for Aeronautics.
Torregrosa, A. J., L. M. Garcia-Cuevas, and P. Quintero. 2021. “On the application of artificial neural network for the development of a nonlinear aeroelastic model.” Aerosp. Sci. Technol. 115 (2021): 106845. https://doi.org/10.1016/j.ast.2021.106845.
Wang, Y., C. Yang, and H. Yang. 2022. “Neural network-based simulation and prediction of precise airdrop trajectory planning.” Aerosp. Sci. Technol. 120 (2022): 107302. https://doi.org/10.1016/j.ast.2021.107302.
Wu, P., W. Yuan, L. Ji, L. Zhou, Z. Zhou, W. Feng, and Y. Guo. 2022. “Missile aerodynamic shape optimization design using deep neural networks.” Aerosp. Sci. Technol. 126 (2022): 107640. https://doi.org/10.1016/j.ast.2022.107640.
Xu, D., Z. Hui, Y. Liu, and G. Chen. 2019. “Morphing control of a new bionic morphing UAV with deep reinforcement learning.” Aerosp. Sci. Technol. 92 (2019): 232–243. https://doi.org/10.1016/j.ast.2019.05.058.
Zhang, T., X. Huo, S. Cen, B. Yang, and G. Zhang. 2018. “Hybrid path planning of a quadrotor UAV based on Q-learning algorithm.” In Proc., 37th Chinese Control Conf. (CCC). New York: IEEE.
Zhang, X., T. Ji, F. Xie, H. Zheng, and Y. Zheng. 2022. “Unsteady flow prediction from sparse measurements by compressed sensing reduced order modeling.” Comput. Methods Appl. Mech. Eng. 393 (2022): 114800. https://doi.org/10.1016/j.cma.2022.114800.
Information & Authors
Information
Published In
Journal of Aerospace Engineering
Volume 37 • Issue 2 • March 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Dec 12, 2022
Accepted: Oct 5, 2023
Published online: Nov 27, 2023
Published in print: Mar 1, 2024
Discussion open until: Apr 27, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.