Reinforcement Learning–Based Adaptive Fault-Tolerant Antidisturbance Control for UAVs Subjected to External Disturbances, Input Uncertainties, and Structural Uncertainties
Publication: Journal of Aerospace Engineering
Volume 38, Issue 1
Abstract
This paper presents a reinforcement learning (RL)–based adaptive fault-tolerant control scheme for quadrotor unmanned aerial vehicles (UAVs) subjected to external disturbances, input uncertainties, and structural uncertainties. In practical engineering, UAV systems often are influenced by aforementioned multiple-source coupled uncertainties, making it challenging to design effective controllers. Herein, first, by introducing a penalty function, a critic network is established to evaluate control performance. Subsequently, the output signals of the critic network are introduced into the updating of actor network, functioning as a reinforcement signal to drive the actor network to approximate unknown nonlinearities. Moreover, an adaptive disturbance boundary estimator is constructed to attenuate the external disturbances and network errors, which are defined collectively in a lumped disturbance set. Additionally, a series of adaptive compensating laws are developed to deal with the input uncertainties. Finally, to tackle multisource coupled uncertainties, a novel RL-based adaptive fault-tolerant controller is proposed which integrates the RL framework, adaptive disturbance boundary estimator, and adaptive input uncertainty compensating laws. Analyzing the Lyapunov function proved that the controlled system is asymptotically stable and all signals are bounded. Numerical simulations revealed the effectiveness and superiority of the proposed method.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
References
Cavanini, L., G. Ippoliti, and E. F. Camacho. 2021. “Model predictive control for a linear parameter varying model of an UAV.” J. Intell. Rob. Syst. 101 (3): 1–18. https://doi.org/10.1007/s10846-021-01337-x.
Chen, C. L. P., G.-X. Wen, Y.-J. Liu, and F.-Y. Wang. 2014. “Adaptive consensus control for a class of nonlinear multiagent time-delay systems using neural networks.” IEEE Trans. Neural Networks Learn. Syst. 25 (6): 1217–1226. https://doi.org/10.1109/TNNLS.2014.2302477.
Chen, J., R. Sun, and B. Zhu. 2020. “Disturbance observer-based control for small nonlinear UAV systems with transient performance constraint.” Aerosp. Sci. Technol. 105 (Oct): 106028. https://doi.org/10.1016/j.ast.2020.106028.
Eliker, K., and W. Zhang. 2020. “Finite-time adaptive integral backstepping fast terminal sliding mode control application on quadrotor UAV.” Int. J. Control Autom. Syst. 18 (Feb): 415–430. https://doi.org/10.1007/s12555-019-0116-3.
Fan, Q.-Y., G.-H. Yang, and D. Ye. 2017. “Quantization-based adaptive actor-critic tracking control with tracking error constraints.” IEEE Trans. Neural Networks Learn. Syst. 29 (4): 970–980. https://doi.org/10.1109/TNNLS.2017.2651104.
Gao, B., Y.-J. Liu, and L. Liu. 2022. “Adaptive neural fault-tolerant control of a quadrotor UAV via fast terminal sliding mode.” Aerosp. Sci. Technol. 129 (Oct): 107818. https://doi.org/10.1016/j.ast.2022.107818.
Gao, B., Y.-J. Liu, and L. Liu. 2023. “Fixed-time neural control of a quadrotor UAV with input and attitude constraints.” IEEE/CAA J. Autom. Sin. 10 (1): 281–283. https://doi.org/10.1109/JAS.2023.123045.
Han, T., Q. Hu, H.-S. Shin, A. Tsourdos, and M. Xin. 2021. “Incremental twisting fault tolerant control for hypersonic vehicles with partial model knowledge.” IEEE Trans. Ind. Inf. 18 (2): 1050–1060. https://doi.org/10.1109/TII.2021.3080303.
Han, T., Q. Hu, and M. Xin. 2020. “Analytical solution of field-of-view limited guidance with constrained impact and capturability analysis.” Aerosp. Sci. Technol. 97 (Feb): 105586. https://doi.org/10.1016/j.ast.2019.105586.
Hu, L., R. Li, T. Xue, and Y. Liu. 2018. “Neuro-adaptive tracking control of a hypersonic flight vehicle with uncertainties using reinforcement synthesis.” Neurocomputing 285 (Apr): 141–153. https://doi.org/10.1016/j.neucom.2018.01.031.
Jiang, H., Y. Xia, R. Hu, D. Ma, and C. Hao. 2019. “A feedback linearization and saturated control structure for quadrotor UAV.” In Proc., 2019 Chinese Control Conf. (CCC), 8277–8282. New York: IEEE.
Li, B., W. Gong, Y. Yang, B. Xiao, and D. Ran. 2021. “Appointed fixed time observer-based sliding mode control for a quadrotor UAV under external disturbances.” IEEE Trans. Aerosp. Electron. Syst. 58 (1): 290–303. https://doi.org/10.1109/TAES.2021.3101562.
Liu, W., X. Cheng, and J. Zhang. 2023. “Command filter-based adaptive fuzzy integral backstepping control for quadrotor UAV with input saturation.” J. Franklin Inst. 360 (1): 484–507. https://doi.org/10.1016/j.jfranklin.2022.10.042.
Liu, Y.-J., S. Li, S. Tong, and C. P. Chen. 2018. “Adaptive reinforcement learning control based on neural approximation for nonlinear discrete-time systems with unknown nonaffine dead-zone input.” IEEE Trans. Neural Networks Learn. Syst. 30 (1): 295–305. https://doi.org/10.1109/TNNLS.2018.2844165.
Lopez-Sanchez, I., and J. Moreno-Valenzuela. 2023. “PID control of quadrotor UAVs: A survey.” Ann. Rev. Control 56 (Jan): 100900. https://doi.org/10.1016/j.arcontrol.2023.100900.
Mao, X., and C. Yuan. 2006. Stochastic differential equations with Markovian switching. London: Imperial College Press.
Mofid, O., S. Mobayen, C. Zhang, and B. Esakki. 2022. “Desired tracking of delayed quadrotor UAV under model uncertainty and wind disturbance using adaptive super-twisting terminal sliding mode control.” ISA Trans. 123 (Apr): 455–471. https://doi.org/10.1016/j.isatra.2021.06.002.
Ouyang, Y., L. Dong, Y. Wei, and C. Sun. 2020. “Neural network based tracking control for an elastic joint robot with input constraint via actor-critic design.” Neurocomputing 409 (Oct): 286–295. https://doi.org/10.1016/j.neucom.2020.05.067.
Peng, Z., J. Hu, K. Shi, R. Luo, R. Huang, B. K. Ghosh, and J. Huang. 2020. “A novel optimal bipartite consensus control scheme for unknown multi-agent systems via model-free reinforcement learning.” Appl. Math. Comput. 369 (Mar): 124821. https://doi.org/10.1016/j.amc.2019.124821.
Polycarpou, M. M., and P. A. Ioannou. 1993. “A robust adaptive nonlinear control design.” In Proc., 1993 American Control Conf., 1365–1369. New York: IEEE.
Pouzesh, M., and S. Mobayen. 2022. “Event-triggered fractional-order sliding mode control technique for stabilization of disturbed quadrotor unmanned aerial vehicles.” Aerosp. Sci. Technol. 121 (Feb): 107337. https://doi.org/10.1016/j.ast.2022.107337.
Qian, M., L. Zhai, G. Zhong, and Z. Gao. 2019. “Adaptive backstepping fault tolerant controller design for UAV with multiple actuator faults.” In Proc., 2019 CAA Symp. on Fault Detection, Supervision and Safety for Technical Processes (SAFEPROCESS), 682–688. New York: IEEE.
Rahmat, M. F., A. Eltayeb, and M. A. M. Basri. 2020. “Adaptive feedback linearization controller for stabilization of quadrotor UAV.” Int. J. Integr. Eng. 12 (4): 1–17.
Shen, S., J. Xu, P. Chen, and Q. Xia. 2023. “Adaptive neural network extended state observer-based finite-time convergent sliding mode control for a quad tiltrotor UAV.” IEEE Trans. Aerosp. Electron. Syst. 59 (5): 6360–6373. https://doi.org/10.1109/TAES.2023.3274733.
Sui, S., C. P. Chen, and S. Tong. 2020. “A novel adaptive NN prescribed performance control for stochastic nonlinear systems.” IEEE Trans. Neural Networks Learn. Syst. 32 (7): 3196–3205. https://doi.org/10.1109/TNNLS.2020.3010333.
Susanto, T., M. B. Setiawan, A. Jayadi, F. Rossi, A. Hamdhi, and J. P. Sembiring. 2021. “Application of unmanned aircraft PID control system for Roll, Pitch and Yaw stability on fixed wings.” In Proc., 2021 Int. Conf. on Computer Science, Information Technology, and Electrical Engineering (ICOMITEE), 186–190. New York: IEEE.
Velagić, J., N. Osmić, V. Klovo, and H. Lačević. 2022. “Design of LQR controller for 3D trajectory tracking of octocopter unmanned aerial vehicle.” In Vol. 1 of Proc., 2022 8th Int. Conf. on Control, Decision and Information Technologies (CoDIT), 63–68. New York: IEEE.
Wang, D., N. Gao, D. Liu, J. Li, and F. L. Lewis. 2024. “Recent progress in reinforcement learning and adaptive dynamic programming for advanced control applications.” IEEE/CAA J. Autom. Sin. 11 (1): 18–36. https://doi.org/10.1109/JAS.2023.123843.
Wang, D., Q. Pan, Y. Shi, J. Hu, and C. Zhao. 2021. “Efficient nonlinear model predictive control for quadrotor trajectory tracking: Algorithms and experiment.” IEEE Trans. Cybern. 51 (10): 5057–5068. https://doi.org/10.1109/TCYB.2020.3043361.
Wei, C., J. Luo, H. Dai, and G. Duan. 2018. “Learning-based adaptive attitude control of spacecraft formation with guaranteed prescribed performance.” IEEE Trans. Cybern. 49 (11): 4004–4016. https://doi.org/10.1109/TCYB.2018.2857400.
Wei, C., J. Luo, H. Dai, Z. Yin, and J. Yuan. 2017. “Low-complexity differentiator-based decentralized fault-tolerant control of uncertain large-scale nonlinear systems with unknown dead zone.” Nonlinear Dyn. 89 (Sep): 2573–2592. https://doi.org/10.1007/s11071-017-3605-z.
Xu, N., X. Liu, Y. Li, G. Zong, X. Zhao, and H. Wang. 2024. “Dynamic event-triggered control for a class of uncertain strict-feedback systems via an improved adaptive neural networks backstepping approach.” IEEE Trans. Autom. Sci. Eng. 1–10. https://doi.org/10.1109/TASE.2024.3374522.
Yang, W., G. Cui, Z. Li, and C. Tao. 2022. “Fuzzy approximation-based adaptive finite-time tracking control for a quadrotor UAV with actuator faults.” Int. J. Fuzzy Syst. 24 (8): 3756–3769. https://doi.org/10.1007/s40815-022-01361-5.
Yu, D., C. L. P. Chen, C.-E. Ren, and S. Sui. 2019. “Swarm control for self-organized system with fixed and switching topology.” IEEE Trans. Cybern. 50 (10): 4481–4494. https://doi.org/10.1109/TCYB.2019.2952913.
Yu, D., M. Yang, Y.-J. Liu, Z. Wang, and C. L. P. Chen. 2022. “Adaptive fuzzy tracking control for uncertain nonlinear systems with multiple actuators and sensors faults.” IEEE Trans. Fuzzy Syst. 31 (1): 104–116. https://doi.org/10.1109/TFUZZ.2022.3182746.
Yu, Z., Y. Zhang, and B. Jiang. 2021. “PID-type fault-tolerant prescribed performance control of fixed-wing UAV.” J. Syst. Eng. Electron. 32 (5): 1053–1061. https://doi.org/10.23919/JSEE.2021.000090.
Yu, Z., Y. Zhang, B. Jiang, X. Yu, J. Fu, Y. Jin, and T. Chai. 2020a. “Distributed adaptive fault-tolerant close formation flight control of multiple trailing fixed-wing UAVs.” ISA Trans. 106 (Nov): 181–199. https://doi.org/10.1016/j.isatra.2020.07.005.
Yu, Z., Y. Zhang, Z. Liu, Y. Qu, C.-Y. Su, and B. Jiang. 2020b. “Decentralized finite-time adaptive fault-tolerant synchronization tracking control for multiple UAVs with prescribed performance.” J. Franklin Inst. 357 (16): 11830–11862. https://doi.org/10.1016/j.jfranklin.2019.11.056.
Zhang, N., J. Xia, J. H. Park, J. Zhang, and H. Shen. 2023. “Improved disturbance observer-based fixed-time adaptive neural network consensus tracking for nonlinear multi-agent systems.” Neural Networks 162 (May): 490–501. https://doi.org/10.1016/j.neunet.2023.03.016.
Zhu, H.-Y., Y.-X. Li, and S. Tong. 2023. “Dynamic event-triggered reinforcement learning control of stochastic nonlinear systems.” IEEE Trans. Fuzzy Syst. 31 (9): 2917–2928. https://doi.org/10.1109/TFUZZ.2023.3235417.
Information & Authors
Information
Published In
Journal of Aerospace Engineering
Volume 38 • Issue 1 • January 2025
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Dec 25, 2023
Accepted: Jul 18, 2024
Published online: Sep 30, 2024
Published in print: Jan 1, 2025
Discussion open until: Feb 28, 2025
ASCE Technical Topics:
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.