Random Prior Network for Autonomous Driving Decision-Making Based on Reinforcement Learning
Publication: Journal of Transportation Engineering, Part A: Systems
Volume 150, Issue 4
Abstract
At present, autonomous driving decision-making solutions take few elements into account while ignoring the unpredictable nature of driving behavior, which makes it challenging to manage complicated traffic situations. To this end, we present a decision-making architecture in this paper that enhances the existing reinforcement learning methodology by combining the bootstrapped technique and the random prior network (RPN). The RPN can give each learner a neural network with unique weights to avoid the contingency created by the artificially built prior functions, while the Bootstrapped technique can balance out the exploration and exploitation. The ego vehicle was trained by three algorithms and verified in random environments to evaluate the effectiveness of our method. The results show that our algorithm outperformed the current reinforcement learning algorithms.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This research was in part supported by the Project of National Natural Science Foundation of China (Nos. 62101314 and 52172371), and partly sponsored by the Program for Shanghai Academic Research Leader (No. 21XD1401100) and the Project of Technical Service Platform for Noise and Vibration Evaluation and Control of New Energy Vehicles (No. 18DZ2295900) at Science and Technology Commission of Shanghai Municipality, China.
References
Agostinelli, F., G. Hocquet, S. Singh, and P. Baldi. 2018. “From reinforcement learning to deep reinforcement learning: An overview.” In Proc., Int. Conf. on Braverman Readings in Machine Learning–Key Ideas from Inception to Current State, 298–328. Cham, Switzerland: Springer.
Alizadeh, A., M. Moghadam, Y. Bicer, N. K. Ure, U. Yavas, and C. Kurtulus. 2019. “Automated lane change decision making using deep reinforcement learning in dynamic and uncertain highway environment.” In Proc., IEEE Intelligent Transportation Systems Conf. (IEEE-ITSC), 1399–1404. New York: IEEE.
Cao, J., X. Wang, Y. Wang, and Y. Tian. 2022. “An improved dueling deep Q-network with optimizing reward functions for driving decision method.” Proc. Inst. Mech. Eng., Part D: J. Automob. Eng. 237 (9): 2295–2309. https://doi.org/10.1177/09544070221106037.
Chae, H., C. M. Kang, B. B. Kim, J. Kim, C. C. Chung, and J. W. Choi. 2017. “Autonomous braking system via deep reinforcement learning.” In Proc., 2017 IEEE 20th Int. Conf. on Intelligent Transportation Systems (ITSC), 1–6. New York: IEEE.
Dann, C., Y. Mansour, M. Mohri, A. Sekhari, and K. Sridharan. 2022. “Guarantees for epsilon-greedy reinforcement learning with function approximation.” In Proc., Machine Learning Research 39th Int. Conf. on Machine Learning, edited by C. Kamalika, J. Stefanie, S. Le, S. Csaba, N. Gang, and S. Sivan, 4666–4689. New York: Proceedings of Machine Learning Research.
Dos Santos, P. H., S. M. Neves, D. O. Sant’Anna, C. H. De Oliveira, and H. D. Carvalho. 2019. “The analytic hierarchy process supporting decision making for sustainable development: An overview of applications.” J. Cleaner Prod. 212 (Jun): 119–138. https://doi.org/10.1016/j.jclepro.2018.11.270.
Feng, D., C. Haase-Schütz, L. Rosenbaum, H. Hertlein, C. Glaeser, F. Timm, W. Wiesbeck, and K. J. I. T. Dietmayer. 2020. “Deep multi-modal object detection and semantic segmentation for autonomous driving: Datasets, methods, and challenges.” IEEE Trans. Intell. Transp. Syst. 22 (3): 1341–1360. https://doi.org/10.1109/TITS.2020.2972974.
Fernandez, N. 2018. Two-stream convolutional networks for end-to-end learning of self-driving cars. Montreal: Advances in Neural Information Processing Systems. https://doi.org/10.48550/arXiv.1811.05785.
Furda, A., and L. Vlacic. 2011. “Enabling safe autonomous driving in real-world city traffic using multiple criteria decision making.” IEEE Intell. Transp. Syst. Mag. 3 (1): 4–17. https://doi.org/10.1109/MITS.2011.940472.
Gao, Z. H., X. T. Yan, F. Gao, and L. He. 2022. “Driver-like decision-making method for vehicle longitudinal autonomous driving based on deep reinforcement learning.” Proc. Inst. Mech. Eng., Part D: J. Automob. Eng. 236 (13): 3060–3070. https://doi.org/10.1177/09544070211063081.
Garaffa, L. C., M. Basso, A. A. Konzen, and E. P. de Freitas. 2021. “Reinforcement learning for mobile robotics exploration: A survey.” IEEE Trans. Neural Networks Learn. Syst. 2021 (Nov): 12. https://doi.org/10.1109/TNNLS.2021.3124466.
Garcia, J., and F. Fernandez. 2015. “A comprehensive survey on safe reinforcement learning.” J. Mach. Learn. Res. 16 (Jun): 1437–1480. https://doi.org/10.5555/2789272.2886795.
Guo, C., X. Wang, L. Su, and Y. Wang. 2021. “Safety distance model for longitudinal collision avoidance of logistics vehicles considering slope and road adhesion coefficient.” Proc. Inst. Mech. Eng., Part D: J. Automob. Eng. 235 (2–3): 498–512. https://doi.org/10.1177/0954407020959744.
Hessel, M., J. Modayil, H. Van Hasselt, T. Schaul, G. Ostrovski, W. Dabney, D. Horgan, B. Piot, M. Azar, and D. Silver. 2018. “Rainbow: Combining improvements in deep reinforcement learning.” In Proc., AAAI Conf. on Artificial Intelligence. Cambridge, MA: MIT Press. https://doi.org/10.1609/aaai.v32i1.11796.
Hoel, C. J., K. Wolff, L. Laine, and IEEE. 2018. “Automated speed and lane change decision making using deep reinforcement learning.” In Proc., 21st IEEE Int. Conf. on Intelligent Transportation Systems (ITSC), 2148–2155. New York: IEEE.
Kamrani, M., A. R. Srinivasan, S. Chakraborty, and A. Khattak. 2020. “Applying Markov decision process to understand driving decisions using basic safety messages data.” Transp. Res. Part C Emerging Technol. 115 (Jun): 102642. https://doi.org/10.1016/j.trc.2020.102642.
Kesting, A., M. Treiber, and D. Helbing. 2007. “General lane-changing model MOBIL for car-following models.” Transp. Res. Rec. 1999 (1): 86–94. https://doi.org/10.3141/1999-10.
Kiran, B. R., I. Sobh, V. Talpaert, P. Mannion, A. A. Al Sallab, S. Yogamani, and P. Perez. 2022. “Deep reinforcement learning for autonomous driving: A survey.” IEEE Trans. Intell. Transp. Syst. 23 (6): 4909–4926. https://doi.org/10.1109/TITS.2021.3054625.
Koh, S., B. Zhou, H. Fang, P. Yang, Z. Yang, Q. Yang, L. Guan, and Z. Ji. 2020. “Real-time deep reinforcement learning based vehicle navigation.” Appl. Soft Comput. 96 (Nov): 106694. https://doi.org/10.1016/j.asoc.2020.106694.
Li, G. F., Y. F. Yang, S. Li, X. D. Qu, N. C. Lyu, and S. E. Li. 2022. “Decision making of autonomous vehicles in lane change scenarios: Deep reinforcement learning approaches with risk awareness.” Transp. Res. Part C Emerging Technol. 134 (Jan): 103452. https://doi.org/10.1016/j.trc.2021.103452.
Lin, Y., J. McPhee, and N. L. Azad. 2020. “Anti-jerk on-ramp merging using deep reinforcement learning.” In Proc., 31st IEEE Intelligent Vehicles Symp. (IV), 7–14. New York: IEEE.
Liu, Y., X. Wang, L. Li, S. Cheng, and Z. Chen. 2019. “A novel lane change decision-making model of autonomous vehicle based on support vector machine.” IEEE Access 7 (Apr): 26543–26550. https://doi.org/10.1109/ACCESS.2019.2900416.
Lopez, P. A., M. Behrisch, L. Bieker-Walz, J. Erdmann, Y.-P. Flötteröd, R. Hilbrich, L. Lücken, J. Rummel, P. Wagner, and E. Wießner. 2018. “Microscopic traffic simulation using sumo.” In Proc., 2018 21st Int. Conf. on Intelligent Transportation Systems (ITSC), 2575–2582. New York: IEEE.
Mnih, V., K. Kavukcuoglu, D. Silver, A. A. Rusu, J. Veness, M. G. Bellemare, A. Graves, M. Riedmiller, A. K. Fidjeland, and G. Ostrovski. 2015. “Human-level control through deep reinforcement learning.” Nature 518 (7540): 529–533. https://doi.org/10.1038/nature14236.
Mousavi, S. S., M. Schukat, and E. Howley. 2018. “Deep reinforcement learning: An overview.” In Proc., SAI Annual Conf. on Areas of Intelligent Systems and Artificial Intelligence and their Applications to the Real World (IntelliSys), 426–440. Cham, Switzerland: Springer.
Osband, I., J. Aslanides, and A. Cassirer. 2018. “Randomized prior functions for deep reinforcement learning.” In Proc., 32nd Conf. on Neural Information Processing Systems (NIPS). Montréal: Advances in Neural Information Processing Systems.
Osband, I., C. Blundell, A. Pritzel, and B. Van Roy. 2016. “Deep exploration via bootstrapped DQN.” Adv. Neural Inf. Process. Syst. 29 (Jun): 1–19. https://doi.org/10.48550/arXiv.1602.04621.
Osband, I., B. Van Roy, D. J. Russo, and Z. Wen. 2019. “Deep exploration via randomized value functions.” J. Mach. Learn. Res. 20 (124): 1–62. https://doi.org/10.48550/arXiv.1703.07608.
Schulman, J., F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. “Proximal policy optimization algorithms.” Preprint, submitted July 20, 2017. https://doi.org/10.48550/arXiv.1707.06347.
Sewak, M. 2019. “Deep q network (DQN), double DQN, and dueling DQN.” In Deep reinforcement learning, 95–108. Berlin: Springer.
Singh, B., R. Kumar, and V. P. Singh. 2021. “Reinforcement learning in robotic applications: A comprehensive survey.” Artif. Intell. Rev. 55 (2): 945–990. https://doi.org/10.1007/s10462-021-09997-9.
Vaidya, O. S., and S. Kumar. 2006. “Analytic hierarchy process: An overview of applications.” Eur. J. Oper. Res. 169 (1): 1–29. https://doi.org/10.1016/j.ejor.2004.04.028.
Wang, J. J., D. B. Zhang, D. B. Zhao, and Y. R. Chen. 2019. “Lane change decision-making through deep reinforcement learning with rule-based constraints.” In Proc., Int. Joint Conf. on Neural Networks (IJCNN). New York: IEEE.
Wolf, P., K. Kurzer, T. Wingert, F. Kuhnt, and J. M. Zollner. 2018. “Adaptive behavior generation for autonomous driving using deep reinforcement learning with compact semantic states.” In Proc., IEEE Intelligent Vehicles Symp. (IV), 993–1000. New York: IEEE.
Wulfmeier, M., D. Z. Wang, and I. Posner. 2016. “Watch this: Scalable cost-function learning for path planning in urban environments.” In Proc., IEEE/RSJ Int. on Intelligent Robots and Systems (IROS), 2089–2095. New York: IEEE.
Xie, J. X., Z. Z. Shao, Y. Li, Y. Guan, and J. D. Tan. 2019. “Deep reinforcement learning with optimized reward functions for robotic trajectory planning.” IEEE Access 7 (Jun): 105669–105679. https://doi.org/10.1109/ACCESS.2019.2932257.
Xu, H., Y. Gao, F. Yu, and T. Darrell. 2017. “End-to-end learning of driving models from large-scale video datasets.” In Proc., IEEE Conf. on Computer Vision and Pattern Recognition, 2174–2182. New York: IEEE.
Yang, W., L. Zheng, Y. N. Li, Y. Ren, and Z. B. Xiong. 2020. “Automated highway driving decision considering driver characteristics.” IEEE Trans. Intell. Transp. Syst. 21 (6): 2350–2359. https://doi.org/10.1109/TITS.2019.2918117.
Ye, F., X. X. Cheng, P. Wang, C. Y. Chan, J. C. Zhang, and IEEE. 2020. “Automated lane change strategy using proximal policy optimization-based deep reinforcement learning.” In Proc., 31st IEEE Intelligent Vehicles Symp. (IV), 1746–1752. New York: IEEE.
Zhu, M., Y. Wang, Z. Pu, J. Hu, X. Wang, and R. Ke. 2020. “Safe, efficient, and comfortable velocity control based on reinforcement learning for autonomous driving.” Transp. Res. Part C Emerging Technol. 117 (Aug): 102662. https://doi.org/10.1016/j.trc.2020.102662.
Zhu, Z., and H. Zhao. 2021. “A survey of deep RL and IL for autonomous driving policy learning.” IEEE Trans. Intell. Transp. Syst. 23 (9): 14043–14065. https://doi.org/10.1109/TITS.2021.3134702.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part A: Systems
Volume 150 • Issue 4 • April 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Nov 16, 2022
Accepted: Nov 16, 2023
Published online: Feb 6, 2024
Published in print: Apr 1, 2024
Discussion open until: Jul 6, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.