Hierarchical Driving Strategy for Connected and Autonomous Vehicles Making a Protected Left Turn at Signalized Intersections
Publication: Journal of Transportation Engineering, Part A: Systems
Volume 149, Issue 3
Abstract
Left-turn execution in autonomous driving at urban intersections is often complex and characterized by unpredicted events, such as vehicles speeding and running a red light. Despite these hazards, autonomous vehicles must drive through intersections safely and efficiently. To solve this problem, a new hierarchical driving strategy (HDS) is proposed for connected and autonomous vehicles making a protected left turn at signalized intersections, which combines the rule-based method and deep reinforcement learning (DRL). The high level of the HDS is the rule-based decision-making module, and the low level is the driving skill, which is dependent on the status. Specifically, the vehicle status when turning left is divided into safe and alert statuses. Further, DRL is used to train the driving skill of the vehicle for each status. The HDS design effectively combines the advantages of end-to-end driving. Moreover, the algorithm was experimentally evaluated in multiple protected left-turn scenarios. Compared with the pure rule-based method, the HDS achieved a 34% reduction in failure rate in the test scenario, and the driving behavior of the autonomous vehicle employing HDS was more intelligent. Moreover, the HDS is highly robust to complex scenarios.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work is supported by the National Natural Science Foundation of China under Grant 51972050 and Heilongjiang Provincial Natural Science Foundation of China under Grant LH2020E006. Besides, sincere thanks to the open-source Carla Simulator, the Roach’s researchers that proposed PPO-Beta and the researchers that proposed simulation model used in our training process.
References
Aradi, S. 2020. “Survey of deep reinforcement learning for motion planning of autonomous vehicles.” IEEE Trans. Intell. Transp. Syst. 23 (2): 1–20. https://doi.org/10.1109/TITS.2020.3024655.
Bojarski, M., et al. 2016. “End to end learning for self-driving cars.” Preprint, submitted April 25, 2016. https://arxiv.org/abs/1604.07316.
Bouton, M., et al. 2019. “Reinforcement learning with probabilistic guarantees for autonomous driving.” Preprint, submitted April 15, 2019. https://arxiv.org/abs/1904.07189.
Caird, J. K., and P. A. Hancock. 2001. “Left turn and gap acceptance crashes.” In Human factors and traffic safety. Tucson, AZ: Lawyers & Judges Publishing Company.
Chen, J., B. Yuan, and M. Tomizuka. 2019. “Model-free deep reinforcement learning for urban autonomous driving.” In Proc., 2019 IEEE Intell. Transp. Syst. Conf. (ITSC), 2765–2771. New York: IEEE.
Dosovitskiy, A., et al. 2017. “CARLA: An open urban driving simulator.” In Vol. 78 of Proc., Conf. on Robot Learning, 1–16. New York: PLMR.
François-Lavet, V., et al. 2018. “An introduction to deep reinforcement learning.” Preprint, submitted November 30, 2018. https://arxiv.org/abs/1811.12560.
Kaempchen, N., B. Schiele, and K. Dietmayer. 2009. “Situation assessment of an autonomous emergency brake for arbitrary vehicle-to-vehicle collision scenarios.” IEEE Trans. Intell. Transp. Syst. 10 (4): 678–687. https://doi.org/10.1109/TITS.2009.2026452.
Kai, S., B. Wang, D. Chen, J. Hao, H. Zhang, and W. Liu. 2020. “A multi-task reinforcement learning approach for navigating unsignalized intersections.” In Proc., 2020 IEEE Intelligent Vehicles Symp. (IV), 1583–1588. New York: IEEE.
Kim, M., S. Lee, J. Lim, J. Choi, and S. G. Kang. 2020. “Unexpected collision avoidance driving strategy using deep reinforcement learning.” IEEE Access 8 (Jan): 17243–17252. https://doi.org/10.1109/ACCESS.2020.2967509.
Lakhal, N. M. B., L. Adouane, O. Nasri, and J. B. H. Slama. 2019. “Risk management for intelligent vehicles based on interval analysis of TTC.” IFAC-Papers On Line 52 (8): 338–343. https://doi.org/10.1016/j.ifacol.2019.08.110.
Li, G., S. Li, S. Li, Y. Qin, D. Cao, X. Qu, and B. Cheng. 2020. “Deep reinforcement learning enabled decision-making for autonomous driving at intersections.” Autom. Innov. 3 (4): 374–385. https://doi.org/10.1007/s42154-020-00113-1.
Liu, T., X. Mu, B. Huang, X. Tang, F. Zhao, X. Wang, and D. Cao. 2020. “Decision-making at unsignalized intersection for autonomous vehicles: Left-turn maneuver with deep reinforcement learning.” Preprint, submitted August 14, 2020. https://arxiv.org/abs/2008.06595.
Liu, Y., Q. Zhang, and D. Zhao. 2021. “A reinforcement learning benchmark for autonomous driving in intersection scenarios.” Preprint, submitted September 22, 2021. https://arxiv.org/abs/2109.10557.
Martinson, M., A. Skrynnik, and A. I. Panov. 2020. “Navigating autonomous vehicle at the road intersection simulator with reinforcement learning.” In Proc., Russian Conf. Artificial Intelligence, 71–84. Berlin: Springer.
Misener, J. A. 2010. Cooperative intersection collision avoidance system (CICAS): Signalized left turn assist and traffic signal adaptation. Berkeley, CA: Univ. of California.
Mnih, V., et al. 2015. “Human-level control through deep reinforcement learning.” Nature 518 (7540): 529–533. https://doi.org/10.1038/nature14236.
Mokhtari, K., and A. R. Wagner. 2021. “Safe deep Q-network for autonomous vehicles at unsignalized intersection.” Preprint, submitted June 8, 2021. https://arxiv.org/abs/2106.04561.
Oh, S., L. Zhang, E. Tseng, W. Williams, H. Kourous, and G. Orosz. 2020. “Safe decision and control of connected automated vehicles for an unprotected left turn.” In Vol. 84270 of Proc., Dynamic Systems and Control Conf., V001T10A005. New York: ASME.
Polders, E., S. Daniels, E. Hermans, T. Brijs, and G. Wets. 2015. “Crash patterns at signalized intersections.” Transp. Res. Rec. 2514 (1): 105–116. https://doi.org/10.3141/2514-12.
Rajendar, S., and V. K. Kaliappan. 2021. “Recent advancements in autonomous emergency braking: A survey.” In Proc., 2021 5th Int. Conf. on Electronics, Communication and Aerospace Technology (ICECA), 1027–1032. New York: IEEE.
Ramezani-Khansari, E., F. M. Nejad, and S. Moogeh. 2020. “Comparing time to collision and time headway as safety criteria.” Pamukkale Üniversitesi Mühendislik Bilimleri Dergisi 27 (6): 669–675.
Schulman, J., P. Moritz, S. Levine, M. Jordan, and P. Abbeel. 2015. “High-dimensional continuous control using generalized advantage estimation.” Preprint, submitted June 8, 2015. https://arxiv.org/abs/1506.02438.
Schulman, J., F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. “Proximal policy optimization algorithms.” Preprint, submitted July 20, 2017. https://arxiv.org/abs/1707.06347.
Seong, H., C. Jung, S. Lee, and D. H. Shim. 2021. “Learning to drive at unsignalized intersections using attention-based deep reinforcement learning.” In Proc., 2021 IEEE Int. Intelligent Transportation Systems Conf. (ITSC). New York: IEEE.
Taleb, T., A. Benslimane, and K. B. Letaief. 2010. “Toward an effective risk-conscious and collaborative vehicular collision avoidance system.” IEEE Trans. Veh. Technol. 59 (3): 1474–1486. https://doi.org/10.1109/TVT.2010.2040639.
Tessler, C., S. Givony, T. Zahavy, D. Mankowitz, and S. Mannor. 2017. “A deep hierarchical approach to lifelong learning in Minecraft.” In Proc., 31st AAAI Conf. Artificial Intelligence. Menlo Park, CA: Association for the Advancement of Artificial Intelligence.
Wisch, M., et al. 2019. “Car-to-car crashes at intersections in Europe and identification of use cases for the test and assessment of respective active vehicle safety systems.” In Proc., 26th Int. Technical Conf. Enhanced Safety Vehicles, 10–13. Washington, DC: Transportation Research Board.
Zhang, Z., A. Liniger, D. Dai, F. Yu, and L. Van Gool. 2021. “End-to-end urban driving by imitating a reinforcement learning coach.” In Proc., IEEE/CVF Int. Conf. on Computer Vision, 15222–15232. New York: IEEE.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part A: Systems
Volume 149 • Issue 3 • March 2023
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Dec 24, 2021
Accepted: Nov 2, 2022
Published online: Dec 20, 2022
Published in print: Mar 1, 2023
Discussion open until: May 20, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.