Performance of State-Shared Multiagent Deep Reinforcement Learning Controlled Signal Corridor with Platooning-Based CAVs
Publication: Journal of Transportation Engineering, Part A: Systems
Volume 149, Issue 8
Abstract
The emerging technologies of connected and automated vehicles (CAVs) and deep reinforcement learning (DRL) provide innovative methods and have a great potential for developing new solutions to improve the efficiency of several intersection systems. Based on the multisource data collected from the transportation environments, CAVs with the cooperative adaptive cruise control (CACC) system could merge into platoons and traverse the intersection quickly and smoothly. Meanwhile, the traffic information about the CAVs enables intelligent traffic signal controls with the help of DRL technologies. This research investigates the performance of a state-shared multiagent deep reinforcement learning (MADRL) controlled signal corridor with platooning-based CAVs. A corridor with seven intersections from the Ingolstadt Traffic Scenario (InTAS) in Germany is selected as a case study. The state information is shared between neighboring intersections to overcome the partial information observation of the decentralized agents in the MADRL framework. A platooning framework with specific CACC systems for the leading and following vehicles is proposed. Results indicate that the state-shared MADRL with CAV platoons could significantly decrease the total waiting time, average queue length, and total emission of the corridor by 80%, 73%, and 54%, respectively, which could be beneficial in further improving the intersection efficiency, designing future intersections, and cooperating signals and CAVs platoons.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or codes that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
The authors want to express their deepest gratitude to the financial support from the United States Department of Transportation and University Transportation Center through the Center for Advanced Multimodal Mobility Solutions and Education (CAMMSE) at The University of North Carolina at Charlotte (Grant No. 69A3551747133).
References
Adebisi, A., Y. Guo, B. Schroeder, J. Ma, B. Cesme, A. Bibeka, and A. Morgan. 2022. “Highway capacity manual capacity adjustment factor development for connected and automated traffic at signalized intersections.” J. Transp. Eng. Part A Syst. 148 (3): 04021121. https://doi.org/10.1061/JTEPBS.0000631.
Alegre, L. N. 2019. “SUMO-RL.” Accessed January 25, 2023. https://github.com/LucasAlegre/sumo-rl.
Ault, J., and G. Sharon. 2021. “Reinforcement learning benchmarks for traffic signal control.” In Proc., NeurIPS 2021. La Jolla, CA: Neural Information Processing Systems Foundation.
Chu, T., J. Wang, L. Codeca, and Z. Li. 2020. “Multi-agent deep reinforcement learning for large-scale traffic signal control.” IEEE Trans. Intell. Transp. Syst. 21 (3): 1086–1095. https://doi.org/10.1109/TITS.2019.2901791.
Harth Michael, L. M., and K. Bogenberger. 2021. “Automated calibration of traffic demand and traffic lights in SUMO using real-world observations.” In Vol. 2 of Proc., SUMO Conf., 133–148. Hanover, Germany: TIB Open Publishing.
Haydari, A., and Y. Yilmaz. 2020. “Deep reinforcement learning for intelligent transportation systems: A survey.” IEEE Trans. Intell. Transp. Syst. 23 (1): 11–32. https://doi.org/10.1109/TITS.2020.3008612.
Liu, H., L. Xiao, X. D. Kan, S. E. Shladover, X. Y. Lu, M. Wang, W. Schakel, and B. van Arem. 2018. Using cooperative adaptive cruise control (CACC) to form high-performance vehicle streams. Berkeley, CA: Univ. of California, Berkeley.
Lobo, S. C., S. Neumeier, E. M. G. Fernandez, and C. Facchi. 2020. “InTAS: The ingolstadt traffic scenario for SUMO.” Preprint, submitted November 4, 2020. http://arxiv.org/abs/2011.11995.
Milanes, V., S. E. Shladover, J. Spring, C. Nowakowski, H. Kawazoe, and M. Nakamura. 2014. “Cooperative adaptive cruise control in real traffic situations.” IEEE Trans. Intell. Transp. Syst. 15 (1): 296–305. https://doi.org/10.1109/TITS.2013.2278494.
Milanés, V., and S. E. Shladover. 2014. “Modeling cooperative and autonomous adaptive cruise control dynamic responses using experimental data.” Transp. Res. Part C: Emerging Technol. 48: 285–300. https://doi.org/10.1016/j.trc.2014.09.001.
Mintsis, E., D. Koutras, K. Porfyri, E. Mitsakis, and L. Lücken. 2019. TransAID Deliverable 3.1-Modelling, simulation and assessment of vehicle automations and automated vehicles’ driver behaviour in mixed traffic-iteration 2. Hannover, Germany: Transition Areas for Infrastructure-Assisted Driving.
Ploeg, J., A. F. A. Serrarens, and G. J. Heijenk. 2011. “Connect & drive: Design and evaluation of cooperative adaptive cruise control for congestion reduction.” J. Mod. Transp. 19 (3): 207–213. https://doi.org/10.1007/BF03325760.
Rajamani, R. 2012. Vehicle dynamics and control: Mechanical engineering series. Boston: Springer.
Segata, M. 2017. “Platooning in SUMO: An open source implementation.” In Proc., SUMO User Conf., 51–62. Hannover, Germany: TIB Open Publishing.
Segata, M., S. Joerer, B. Bloessl, C. Sommer, F. Dressler, and R. Lo Cigno. 2014. “Plexe: A platooning extension for Veins.” In Proc., 2014 IEEE Vehicular Networking Conf., 53–60. New York: IEEE.
Shi, S., and F. Chen. 2018. “Deep recurrent Q-learning method for area traffic coordination control.” J. Adv. Math. Comput. Sci. 27 (3): 1–11. https://doi.org/10.9734/JAMCS/2018/41281.
Shladover, S. E. 2018. “Connected and automated vehicle systems: Introduction and overview.” J. Intell. Transp. Syst. Technol. Plann. Oper. 22 (3): 190–200. https://doi.org/10.1080/15472450.2017.1336053.
Shladover, S. E., D. Su, and X. Y. Lu. 2012. “Impacts of cooperative adaptive cruise control on freeway traffic flow.” Transp. Rec. J. Transp. Res. 2324 (1): 63–70. https://doi.org/10.3141/2324-08.
Song, L., and W. Fan. 2021. “Traffic signal control under mixed traffic with connected and automated vehicles: A transfer-based deep reinforcement learning approach.” IEEE Access 9: 145228–145237. https://doi.org/10.1109/ACCESS.2021.3123273.
Song, L., W. Fan, and P. Liu. 2021. “Exploring the effects of connected and automated vehicles at fixed and actuated signalized intersections with different market penetration rates.” Transp. Plan. Technol. 44 (6): 577–593. https://doi.org/10.1080/03081060.2021.1943129.
SUMO. 2022. “Traffic lights: SUMO documentation.” Accessed January 25, 2023. https://sumo.dlr.de/docs/Simulation/Traffic_Lights.html#coordination.
Treiber, M., A. Hennecke, and D. Helbing. 2000. “Congested traffic states in empirical observations and microscopic simulations.” Phys. Rev. E 62 (2): 1805–1824. https://doi.org/10.1103/PhysRevE.62.1805.
Treiber, M., and A. Kesting. 2013. Traffic flow dynamics: Data, models and simulation. Berlin: Springer.
Virdi, N., H. Grzybowska, S. T. Waller, and V. Dixit. 2019. “A safety assessment of mixed fleets with connected and autonomous vehicles using the surrogate safety assessment module.” Accid. Anal. Prev. 131 (Oct): 95–111. https://doi.org/10.1016/j.aap.2019.06.001.
Xiao, L., M. Wang, and B. Van Arem. 2017. “Realistic car-following models for microscopic simulation of adaptive and cooperative adaptive cruise control vehicles.” Transp. Rec. J. Transp. Res. 2623 (1): 1–9. https://doi.org/10.3141/2623-01.
Yang, H., H. Rakha, and M. V. Ala. 2017. “Eco-cooperative adaptive cruise control at signalized intersections considering queue effects.” IEEE Trans. Intell. Transp. Syst. 18 (6): 1575–1585. https://doi.org/10.1109/TITS.2016.2613740.
Zhang, R., A. Ishikawa, W. Wang, B. Striner, and O. K. Tonguz. 2021. “Using reinforcement learning with partial vehicle detection for intelligent traffic signal control.” IEEE Trans. Intell. Transp. Syst. 22 (1): 404–415. https://doi.org/10.1109/TITS.2019.2958859.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part A: Systems
Volume 149 • Issue 8 • August 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Oct 24, 2022
Accepted: Mar 31, 2023
Published online: Jun 2, 2023
Published in print: Aug 1, 2023
Discussion open until: Nov 2, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.