Trajectory Tracking Control for Fixed-Wing UAV Based on DDPG
Publication: Journal of Aerospace Engineering
Volume 37, Issue 3
Abstract
The study proposes a method for the trajectory tracking control of a fixed-wing unmanned aerial vehicle (UAV) based on the deep deterministic policy gradient (DDPG). First, the problem of controlling the trajectory of a fixed-wing UAV is combined with the reinforcement learning framework and transformed into a Markov decision process, and a DDPG agent is established in the framework of TensorFlow. Second, we conducted simulations to train and optimize the model in a 3D environment of trajectory tracking control and obtained an integrated DDPG-based trajectory tracking controller that can regulate functions ranging from the state of flight of the UAV to rudder control. Third, we constructed a digital simulation system to verify the proposed method while considering the influence of parametric uncertainties, measurement-induced noise, and delays in the response of the control system. The effectiveness and robustness of the proposed DDPG controller were verified by comparing its performance with that of traditional proportional-integral-derivative (PID) control.
Practical Applications
Unmanned aerial vehicles have gained widespread use across numerous sectors of society. However, the control process for fixed-wing UAVs is more intricate and challenging than that for rotor type UAVs due to the coupled and nonlinear nature of their dynamic models. To address this issue, decoupling and splitting hierarchical control methods are commonly employed. However, these methods can introduce errors and affect flight performance. In contrast, artificial intelligence, particularly deep learning and reinforcement learning, has made significant advancements and found successful applications in areas such as robot control and gaming. To address the challenges posed by complex forms, excessive measurement information, and manual parameter tuning associated with traditional control approaches for fixed-wing UAVs, this paper introduces a deep reinforcement learning algorithm, i.e., the deep deterministic policy gradient, into the control process of fixed-wing UAVs. The paper presents a trajectory tracking control policy for UAVs, which efficiently achieves the task of tracking a desired trajectory without the need for selecting multiple working state nodes or designing linear control laws. Further, through simulation verification, the proposed policy demonstrates adaptability to interference caused by parameter bias, measurement noise, and control system response delays while achieving control accuracy comparable with classical control methods.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Code and models that support the findings of this study are available from the corresponding author upon reasonable request.
References
Bohn, E., M. C. Erlend, S. Moe, and T. C. Johansen. 2019. “Deep reinforcement learning attitude control of fixed-wing UAVs using proximal policy optimization.” In Proc., 2019 Int. Conf. on Unmanned Aircraft Systems (ICUAS), 523–533. New York: IEEE. https://doi.org/10.1109/ICUAS.2019.8798254.
Brigido-González, J. D., and H. Rodríguez-Cortés. 2016. “Experimental validation of an adaptive total energy control system strategy for the longitudinal dynamics of a fixed-wing aircraft.” J. Aerosp. Eng. 29 (1): 04015024. https://doi.org/10.1061/(ASCE)AS.1943-5525.0000512.
Bulka, E., and M. Nahon. 2019. “Automatic control for aerobatic maneuvering of agile fixed-wing UAVs.” J. Intell. Robot. Syst. 93 (1): 85–100. https://doi.org/10.1007/s10846-018-0790-z.
Clarke, S. G., and S. Hwang. 2020. “Deep reinforcement learning control for aerobatic maneuvering of agile fixed-wing aircraft.” In Proc., AIAA Scitech 2020 Forum, 6–10. Reston, VA: American Institute of Aeronautics and Astronautics. https://doi.org/10.2514/6.2020-0136.
Farrell, J., M. Sharma, and M. Polycarpou. 2005. “Back-stepping based flight control with adaptive function approximation.” J. Guid. Control Dyn. 28 (6): 1089–1102. https://doi.org/10.2514/1.13030.
Fujimoto, S., H. V. Hoof, and D. Meger. 2018. “Addressing function approximation error in actor-critic methods.” Preprints, submitted February 26, 2018. http://arxiv.org/abs/1802.09477.
He, S., C. Lee, H. S. Shin, and A. Tsourdos. 2019a. “Minimum-effort waypoint following code.” J. Guid. Control Dyn. 42 (7): 1551–1561. https://doi.org/10.2514/1.G004045.
He, S., H. S. Shin, A. Tsourdos, and C. Lee. 2019b. “Energy-optimal waypoint-following guidance considering autopilot dynamics.” IEEE Trans. Aerosp. Electron. Syst. 56 (4): 2701–2717. https://doi.org/10.1109/TAES.2019.2954149.
Huang, X., et al. 2021. “The Tianwen-1 guidance, navigation, and control for mars entry, descent, and landing.” Space Sci. Technol. 2021: 9846185. https://doi.org/10.34133/2021/9846185.
Huang, X., W. Luo, and J. Liu. 2019. “Attitude control of fixed-wing UAV based on DDQN.” In Proc., 2019 Chinese Automation Congress (CAC), 4722–4726. New York: IEEE. https://doi.org/10.1109/CAC48633.2019.8996970.
Jiang, S., C. Liu, and Y. Gao. 2021. “MIMO adaptive high-order sliding mode control for quadrotor attitude tracking.” J. Aerosp. Eng. 34 (4): 04021022. https://doi.org/10.1061/(ASCE)AS.1943-5525.0001271.
Li, K., Z. Bai, H. S. Shin, A. Tsourdos, and MJ. Tahk. 2021. “Capturability of 3D RTPN guidance law against true-arbitrarily maneuvering target with maneuverability limitation.” Chin. J. Aeronaut. 35 (7): 75–90. https://doi.org/10.1016/j.cja.2021.10.004.
Li, K., H. S. Shin, A. Tsourdos, and MJ. Tahk. 2020. “Performance of 3D PPN against arbitrarily maneuvering target for homing phase.” IEEE Trans. Aerosp. Electron. Syst. 56 (5): 3878–3891. https://doi.org/10.1109/TAES.2020.2987404.
Li, K., W. Su, and L. Chen. 2018. “Performance analysis of three-dimensional differential geometric guidance law against low-speed maneuvering targets.” Astrodynamics 2 (3): 233–247. https://doi.org/10.1007/s42064-018-0023-z.
Lillicrap, T. P., J. J. Hunt, A. Pritzel, N. Heess, T. Erez, Y. Tassa, D. Silver, and D. Wierstra. 2016. “Continuous control with deep reinforcement learning.” Comput. Sci. 8 (6): 187. https://doi.org/10.48550/arXiv.1509.02971.
Lu, P. 2017. “Introducing computational guidance and control.” J. Guid. Control Dyn. 40 (2): 193. https://doi.org/10.2514/1.G002745.
Mnih, V., et al. 2015. “Human level control through deep reinforcement learning.” Nature 518 (7540): 529–533. https://doi.org/10.1038/nature14236.
Pei, P., S. He, J. Wang, and D. Lin. 2021. “An integrated guidance and control algorithm for deep reinforcement learning.” J. Astronaut. 42 (10): 1293–1304. https://doi.org/10.3873/j.issn.1000-1328.2021.10.010.
Qiu, X., C. Gao, K. Wang, and W. Jing. 2021. “Attitude control of a moving mass–actuated UAV based on deep reinforcement learning.” J. Aerosp. Eng. 35 (2): 04021133. https://doi.org/10.1061/(ASCE)AS.1943-5525.0001381.
Randal, W. B., and W. M. Timonthy. 2012. Small unmanned aircraft: Theory and practice. Princeton, NJ: Princeton University Press.
Robert, F. S. 2004. Flight dynamics. Princeton, NJ: Princeton University Press.
Snell, S. A., D. F. Enns, and W. L. Garrard. 2015. “Nonlinear inversion flight control for a supermaneuverable aircraft.” J. Guid. Control Dyn. 15 (4): 976–984. https://doi.org/10.2514/3.20932.
Song, X., Y. Wang, Z. Cai, J. Zhao, X. Chen, and D. Song. 2020. “UAV landing trajectory tracking control based on deep reinforcement learning.” Aeronaut. Sci. Technol. 31 (1): 68–75.
Sutton, R. S., and A. G. Barto. 1998. “Reinforcement learning: An introduction.” IEEE Trans. Neural Networks 9 (5): 1054. https://doi.org/10.1109/TNN.1998.712192.
Zhang, H., J. Li, Z. Wang, and Y. Guan. 2021a. “Guidance navigation and control for Chang’E-5 powered descent.” Space Sci. Technol. 2021: 9823609. https://doi.org/10.34133/2021/9823609.
Zhang, S., X. Du, J. Xiao, and J. Huang. 2022. “Six-degree-of-freedom flight intelligent control of fixed-wing aircraft based on deep reinforcement learning.” J. Command Control 8 (2): 179–188.
Zhang, S., X. Du, Z. Xiao, J. Huang, and K. He. 2021b. “Reinforcement learning control for 6 DOF flight of fixed-wing aircraft.” In Proc., 2021 33rd Chinese Control and Decision Conf., 5454–5460. New York: IEEE. https://doi.org/10.1109/CCDC52312.2021.9602605.
Zhang, S., J. Xiao, J. Huang, and X. Du. 2020. “Quadrotor flight control based on deep reinforcement learning.” In Proc., China 2020 Conf. of Flight Mechanics and Test. New York: IEEE.
Zhao, S., X. Wang, D. Zhang, and L. Shen. 2017. “Model-free fuzzy adaptive control of the heading angle of fixed-wing unmanned aerial vehicles.” J. Aerosp. Eng. 30 (4): 04017019. https://doi.org/10.1061/(ASCE)AS.1943-5525.0000730.
Zhen, Y., M. Hao, and W. Sun. 2020. “Deep reinforcement learning attitude control of fixed-wing UAVs.” In Proc., 2020 3rd Int. Conf. on Unmanned Systems (ICUS), 239–243. New York: IEEE. https://doi.org/10.1109/ICUS50048.2020.9274875.
Information & Authors
Information
Published In
Copyright
© 2024 American Society of Civil Engineers.
History
Received: May 28, 2023
Accepted: Nov 22, 2023
Published online: Jan 31, 2024
Published in print: May 1, 2024
Discussion open until: Jun 30, 2024
ASCE Technical Topics:
- Aerospace engineering
- Aircraft and spacecraft
- Aircraft wings
- Detection methods
- Engineering fundamentals
- Equipment and machinery
- Markov process
- Mathematics
- Methodology (by type)
- Models (by type)
- Optimization models
- Probability
- Research methods (by type)
- Stochastic processes
- Three-dimensional models
- Tracking
- Unmanned vehicles
- Verification
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.