Variable Speed Limit Control for Mixed Traffic Flow on Highways Based on Deep-Reinforcement Learning
Publication: Journal of Transportation Engineering, Part A: Systems
Volume 150, Issue 3
Abstract
With the development of autonomous driving, a mixed traffic flow state composed of connected automated vehicles (CAVs) and human-driven vehicles (HVs) will last for an extended period. The abundant computing resources and CAVs with high compliance in the intelligent connected environment provide a good situation for variable speed limit control on highways, which helps even the traffic flow and improves traffic efficiency and safety. In this paper, we propose a variable speed limit control method for mixed traffic flow based on deep-reinforcement learning. First, the variable speed limit control problem is abstracted into a Markov decision process and the factors of real-time CAV penetration rates and predictions are considered in the state description. Different from variable message signs (VMS), CAVs are taken as the executive objects of the controller so that the variable speed limit control for mixed traffic flow is realized indirectly through the interaction with HVs. Next, double deep Q network (DDQN) is introduced to calculate the optimal speed limit in different states. Finally, the empirical study on US101-S proves the effectiveness of the proposed model. The results show that the variable speed limit control model based on the DDQN algorithm can effectively improve the efficiency and environmental benefits of mixed traffic flow. Moreover, the multi-objective reward function can achieve a better control effect than the single objective. Besides, the proposed model outperforms other models in this paper and predictive factors can further improve proactive control performance. In addition, with the increasing penetration of CAV, the proposed model achieves a better control effect.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This work was supported by National Natural Science Foundation of China Grant No. 52072143.
References
Ahmic, K., A. Tahirbegovic, A. Tahirovic, D. Watzenig, and G. Stettinger. 2020. “Simulation framework for platooning based on gazebo and SUMO.” In Proc., 2020 IEEE 3rd Connected and Automated Vehicles Symposium (CAVs), New York: IEEE.
Allaby, P., B. Hellinga, and M. Bullock. 2007. “Variable speed limits: Safety and operational impacts of a candidate control strategy for freeway applications.” IEEE Trans. Intell. Transp. Syst. 8 (4): 671–680. https://doi.org/10.1109/TITS.2007.908562.
Arora, K., and L. Kattan. 2023. “Operational and safety impacts of integrated variable speed limit with dynamic hard shoulder running.” J. Intell. Transport. Syst. 27 (6), 769–798. https://doi.org/10.1080/15472450.2022.2078664.
Bouchemal, N., and S. Kallel. 2021. “Testbed of V2X infrastructure for autonomous vehicles.” Ann. Telecommun. 76 (9–10): 731–743. https://doi.org/10.1007/s12243-021-00880-w.
Chanfreut, P., J. M. Maestre, and E. F. Camacho. 2021. “Coalitional model predictive control on freeways traffic networks.” IEEE Trans. Intell. Transp. Syst. 22 (11): 6772–6783. https://doi.org/10.1109/TITS.2020.2994772.
Fang, X., and T. Tettamanti. 2021. “Traffic congestion phenomena when motorway meets urban road network.” In Proc., 2021 IEEE 25th Int. Conf. on Intelligent Engineering Systems (INES), 000025–000030. New York: IEEE.
Frejo, J. R. D., I. Papamichail, M. Papageorgiou, and B. D. Schutter. 2018. “A new macroscopic model for variable speed limits.” IFAC-PapersOnLine 51 (9): 343–348. https://doi.org/10.1016/j.ifacol.2018.07.056.
Gao, H., H. Jia, and L. Yang. 2022. “An improved CEEMDAN-FE-TCN model for highway traffic flow prediction.” J. Adv. Transp. 2022 (May): 2265000. https://doi.org/10.1155/2022/2265000.
Geistefeldt, J. 2011. “Capacity effects of variable speed limits on German freeways.” Procedia-Soc. Behav. Sci. 16 (Jan): 48–56. https://doi.org/10.1016/j.sbspro.2011.04.428.
Han, Y., M. Wang, Z. He, Z. Li, H. Wang, and P. Liu. 2021. “A linear Lagrangian model predictive controller of macro- and micro- variable speed limits to eliminate freeway jam waves.” Transp. Res. Part C Emerging Technol. 128 (Jul): 103121. https://doi.org/10.1016/j.trc.2021.103121.
Hasan, T., M. Abdel-Aty, and N. Mahmoud. 2023. “Freeway crash prediction models with variable speed limit/variable advisory speed.” J. Transp. Eng. Part A. Syst. 149 (3): 04022159. https://doi.org/10.1061/JTEPBS.TEENG-7349.
Hu, F., and W. Huang. 2022. “A stage pressure-based adaptive traffic signal control using reinforcement learning.” In Proc., Int. Conf. on Intelligent Traffic Systems and Smart City (ITSSC 2021), edited by G. Tan and F. Cen, 121651. Bellingham, WA: SPIE-International Society for Optical Engineering.
Jia, D., D. Ngoduy, and H. L. Vu. 2019. “A multiclass microscopic model for heterogeneous platoon with vehicle-to-vehicle communication.” Transportmetrica B: Trans. Dyn. 7 (1): 311–335. https://doi.org/10.1080/21680566.2018.1434021.
Khondaker, B., and L. Kattan. 2015. “Variable speed limit: A microscopic analysis in a connected vehicle environment.” Transp. Res. Part C Emerging Technol. 58 (Mar): 146–159. https://doi.org/10.1016/j.trc.2015.07.014.
Lu, Q., L. Tettamanti, D. Horcher, and I. Varga. 2020a. “The impact of autonomous vehicles on urban traffic network capacity: An experimental analysis by microscopic traffic simulation.” Transp. Lett. 12 (8): 540–549. https://doi.org/10.1080/19427867.2019.1662561.
Lu, W.-Q., Y.-K. Rui, B. Ran, and Y.-L. Gu. 2020b. “Traffic flow prediction based on hybrid deep learning under connected and automated vehicle environment.” J. Transp. Syst. Eng. Inf. Technol. 20 (3): 47–53. https://doi.org/10.16097/j.cnki.1009-6744.2020.03.008.
Maiti, N., and B. R. Chilukuri. 2021. “Traffic signal control for an isolated intersection using reinforcement learning.” In Proc., 2021 Int. Conf. on Communication Systems & Networks (comsnets), 629–633. New York: IEEE.
Mao, P., X. Ji, X. Qu, L. Li, and B. Ran. 2022. “A variable speed limit control based on variable cell transmission model in the connecting traffic environment.” IEEE Trans. Intell. Transp. Syst. 23 (10): 17632–17643. https://doi.org/10.1109/TITS.2022.3160374.
Milanes, V., S. E. Shladover, J. Spring, C. Nowakowski, H. Kawazoe, and M. Nakamura. 2014. “Cooperative adaptive cruise control in real traffic situations.” IEEE Trans. Intell. Transp. Syst. 15 (1): 296–305. https://doi.org/10.1109/TITS.2013.2278494.
Milanés, V., and S. E. Shladover. 2014. “Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data.” Transp. Res. Part C Emerging Technol. 48 (Mar): 285–300. https://doi.org/10.1016/j.trc.2014.09.001.
Mnih, V., et al. 2015. “Human-level control through deep reinforcement learning.” Nature 518 (7540): 529–533. https://doi.org/10.1038/nature14236.
Müller, E. R., R. C. Carlson, and W. Kraus. 2016. “Cooperative mainstream traffic flow control on freeways.” IFAC-PapersOnLine 49 (32): 89–94. https://doi.org/10.1016/j.ifacol.2016.12.195.
Nie, W., Y. You, V. C. S. Lee, and Y. Duan. 2021. “Variable speed limit control for individual vehicles on freeway bottlenecks with mixed human and automated traffic flows.” In Proc., 2021 IEEE Int. Intelligent Transportation Systems Conf. (ITSC), 2492–2498. New York: IEEE.
Papageorgiou, M., E. Kosmatopoulos, and I. Papamichail. 2008. “Effects of variable speed limits on motorway traffic flow.” Transp. Res. Rec. 2047 (1): 37–48. https://doi.org/10.3141/2047-05.
Piao, J., and M. McDonald. 2008. “Safety impacts of variable speed limits–A simulation study.” In Proc., 2008 11th Int. IEEE Conf. on Intelligent Transportation Systems, 833–837. New York: IEEE.
Qin, Y.-Y., H. Wang, W. Wang, and Q. Wan. 2017a. “Fundamental diagram model of heterogeneous traffic flow mixed with cooperative adaptive cruise control vehicles and adaptive cruise control vehicles.” China J. Highway Transp. 30 (10): 127–136.
Qin, Y.-Y., H. Wang, W. Wang, and Q. Wan. 2017b. “Stability analysis and fundamental diagram of heterogeneous traffic flow mixed with cooperative adaptive cruise control vehicles.” Acta Phys. Sin. 66 (9). https://doi.org/10.7498/aps.66.094502.
Qu, Z., H. Li, Z. Li, and T. Zhong. 2022. “Short-term traffic flow forecasting method with M-B-LSTM hybrid network.” IEEE Trans. Intell. Transp. Syst. 23 (1): 225–235. https://doi.org/10.1109/TITS.2020.3009725.
Silgu, M. A., İ. G. Erdağı, G. Göksu, and H. B. Celikoglu. 2022. “Combined control of freeway traffic involving cooperative adaptive cruise controlled and human driven vehicles using feedback control through SUMO.” IEEE Trans. Intell. Transp. Syst. 23 (8): 11011–11025. https://doi.org/10.1109/TITS.2021.3098640.
Song, L., and W. Fan. 2021. “Traffic signal control under mixed traffic with connected and automated vehicles: A transfer-based deep reinforcement learning approach.” IEEE Access 9 (Oct): 145228–145237. https://doi.org/10.1109/ACCESS.2021.3123273.
Treiber, M., A. Hennecke, and D. Helbing. 2000. “Congested traffic states in empirical observations and microscopic simulations.” Phys. Rev. E: Stat. Phys. Plasmas Fluids Relat. Interdiscip. Top. 62 (2A): 1805–1824. https://doi.org/10.1103/PhysRevE.62.1805.
Van Hasselt, H. V., A. Guez, and D. Silver. 2015. “Deep reinforcement learning with double Q-learning.” In Proc., AAAI Conf. on Artificial Intelligence. Reston, VA: American Institute of Aeronautics and Astronautics.
Vrbanić, F., E. Ivanjko, S. Mandžuka, and M. Miletić. 2021. “Reinforcement learning based variable speed limit control for mixed traffic flows.” In Proc., 2021 29th Mediterranean Conf. on Control and Automation (MED), 560–565. New York: IEEE. https://doi.org/10.1109/MED51440.2021.9480215.
Wang, X., J. Yang, L. Zhang, and P. Wang. 2021. “Modeling and simulating of single autonomous vehicle under urban conventional traffic flow.” In Proc., 2021 IEEE 16th Conf. on Industrial Electronics and Applications (ICIEA 2021), 1872–1876. New York: IEEE.
Wu, Y., H. Tan, L. Qin, and B. Ran. 2020. “Differential variable speed limits control for freeway recurrent bottlenecks via deep actor-critic algorithm.” Transp. Res. Part C Emerging Technol. 117 (4): 102649. https://doi.org/10.1016/j.trc.2020.102649.
Wu, Y., H. Tan, L. Qin, B. Ran, and Z. Jiang. 2018. “A hybrid deep learning based traffic flow prediction method and its understanding.” Transp. Res. Part C: Emerging Technol. 90 (May): 166–180. https://doi.org/10.1016/j.trc.2018.03.001.
Xiao, D., S. Kang, X. Xu, and Z. Shen. 2022. “Reinforcement learning based mainline dynamic speed limit adjustment of expressway off-ramp upstream under connected and autonomous vehicles environment.” IET Intel. Transp. Syst. 16 (12): 1809–1819. https://doi.org/10.1049/itr2.12225.
Xie, D. F., X. M. Zhao, and Z. He. 2018. “Heterogeneous traffic mixing regular and connected vehicles: Modeling and stabilization.” IEEE Trans. Intell. Transp. Syst. 20 (6): 2060–2071. https://doi.org/10.1109/TITS.2018.2857465.
Yu, J., A. Arab, J. Yi, X. Pei, and X. Guo. 2023. “Hierarchical framework integrating rapidly-exploring random tree with deep reinforcement learning for autonomous vehicle.” Appl. Intell. 53 (13): 16473–16486. https://doi.org/10.1007/s10489-022-04358-7.
Yu, M., and W. D. Fan. 2019. “Optimal variable speed limit control in connected autonomous vehicle environment for relieving freeway congestion.” J. Transp. Eng. 145 (4): 04019007. https://doi.org/10.1061/JTEPBS.0000227.
Zhang, C., E. Chung, N. R. Sabar, A. Bhaskar, and Y. Ma. 2023. “Optimisation of variable speed limits at the freeway lane drop bottleneck.” Transportmetrica A: Transport Sci. 19 (2): 2033878. https://doi.org/10.1080/23249935.2022.2033878.
Zhang, L., M. Zhang, J. Ma, and J. Ge. 2021. “A data-driven control strategy for urban express ramp.” Comput. Intell. Neurosci. 2021 (Nov): 6540972. https://doi.org/10.1155/2021/6540972.
Zhang, L. Y., X. K. Duan, J. Ma, M. Zhang, Y. Wen, and Y. Wang. 2022. “Mechanism of road capacity under different penetration scenarios of autonomous vehicles.” Int. J. Simul. Modell. 21 (1): 172–183. https://doi.org/10.2507/IJSIMM21-1-CO4.
Zhao, H., J. Zhan, and L. Zhang. 2023. “Saturated boundary feedback stabilization for LWR traffic flow model.” Syst. Control Lett. 173 (Mar): 105465. https://doi.org/10.1016/j.sysconle.2023.105465.
Zheng, N., J. Li, Z. Mao, and K. Tei. 2022. “From local to global: A curriculum learning approach for reinforcement learning-based traffic signal control.” In Proc., 2022 2nd IEEE Int. Conf. on Software Engineering and Artificial Intelligence (SEAI 2022), 253–258. New York: IEEE.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part A: Systems
Volume 150 • Issue 3 • March 2024
Copyright
© 2023 American Society of Civil Engineers.
History
Received: May 30, 2023
Accepted: Oct 11, 2023
Published online: Dec 30, 2023
Published in print: Mar 1, 2024
Discussion open until: May 30, 2024
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.