Real-Time Optimization of Urban Rail Transit Train Scheduling via Advantage Actor–Critic Deep Reinforcement Learning
Publication: Journal of Transportation Engineering, Part A: Systems
Volume 150, Issue 9
Abstract
Under the condition of urban rail transit uncertainty of passenger demand and the high frequency of departure intervals, this study presents an innovative real-time urban rail transit (URT) train service scheduling control framework. In the context of a bidirectional urban rail transit line, a high-fidelity urban rail transit simulation environment was constructed. Within this environment, an advantage actor–critic (A2C) reinforcement learning approach was utilized to train a suitable strategy aimed at minimizing both passenger waiting costs and transit authority operational expenses. Subject to specific constraints, the strategy is designed to generate real-time train schedule based on the representation of traffic state using station congestion levels and train positions. Experimental results on Lines 3 and S7 of Nanjing Metro demonstrated the agent’s effectiveness in achieving high-performance schedules across various scenarios. This research integrates deep reinforcement learning into the optimization of dynamic traffic systems, showing great potential for enhancing the efficiency and resilience of urban transport systems.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code used during the study were provided by a third party. Direct requests for these materials may be made to the provider as indicated in the Acknowledgements.
Acknowledgments
The authors express their heartfelt gratitude to Nanjing Metro for generously providing the data support for this research.
References
Bertsekas, D. 2019. Reinforcement learning and optimal control. Nashua, NH: Athena Scientific.
Chang, C. S., and C. M. Kwan. 2004. “Evaluation of evolutionary algorithms for multi-objective train schedule optimization.” In Proc., Australasian Joint Conf. on Artificial Intelligence, 803–815. Berlin: Springer.
Chu, W., X. Zhang, J. Chen, and X. Sun. 2016. “Pareto optimal train scheduling for urban rail transit using generalized particle swarm optimization.” Adv. Mech. Eng. 8 (10): 168781401667242. https://doi.org/10.1177/1687814016672427.
Guo, Y., C. Zhang, C. Wu, and S. Lu. 2021. “Multiagent system–based near-real-time trajectory and microscopic timetable optimization for rail transit network.” J. Transp. Eng. Part A. Syst. 147 (2): 04020153. https://doi.org/10.1061/JTEPBS.0000473.
He, D., S. Guo, Y. Chen, B. Liu, J. Chen, and W. Xiang. 2021. “Energy efficient metro train running time rescheduling model for fully automatic operation lines.” J. Transp. Eng. Part A. Syst. 147 (7): 04021032. https://doi.org/10.1061/JTEPBS.0000546.
Jamili, A., M. A. Shafia, S. J. Sadjadi, and R. Tavakkoli-Moghaddam. 2012. “Solving a periodic single-track train timetabling problem by an efficient hybrid algorithm.” Eng. Appl. Artif. Intell. 25 (4): 793–800. https://doi.org/10.1016/j.engappai.2012.01.020.
Jiang, M., A. Wang, Z. Li, and F. Tsung. 2023. “A unified probabilistic framework for spatiotemporal passenger crowdedness inference within urban rail transit network.” Preprint, submitted June 14, 2023. https://arxiv.org/abs/2306.08343.
Kroon, L. G., and L. W. Peeters. 2003. “A variable trip time model for cyclic railway timetabling.” Transp. Sci. 37 (2): 198–212. https://doi.org/10.1287/trsc.37.2.198.15247.
Li, D., X. Dong, J. Cao, S. Zhang, and L. Yang. 2022. “Energy-efficient rail transit vertical alignment optimization: Gaussian pseudospectral method.” J. Transp. Eng. Part A. Syst. 148 (1): 04021100. https://doi.org/10.1061/JTEPBS.0000590.
Li, W., and S. Ni. 2022. “Train timetabling with the general learning environment and multi-agent deep reinforcement learning.” Transp. Res. Part B. Methodol. 157 (Mar): 230–251. https://doi.org/10.1016/j.trb.2022.02.006.
Liebchen, C., M. Proksch, and F. H. Wagner. 2008. “Performance of algorithms for periodic timetable optimization.” In Proc., Computer-Aided Systems in Public Transport, 151–180. Berlin: Springer.
Lu, J., G. Ren, and L. Xu. 2020. “Analysis of subway station distribution capacity based on automatic fare collection data of Nanjing Metro.” J. Transp. Eng. Part A. Syst. 146 (2): 04019067. https://doi.org/10.1061/JTEPBS.0000304.
Niu, H., and X. Zhou. 2013. “Optimizing urban rail timetable under time-dependent demand and oversaturated conditions.” Transp. Res. Part C Emerging Technol. 36 (Nov): 212–230. https://doi.org/10.1016/j.trc.2013.08.016.
Odijk, M. A. 1996. “A constraint generation algorithm for the construction of periodic railway timetables.” Transp. Res. Part B. Methodol. 30 (6): 455–464. https://doi.org/10.1016/0191-2615(96)00005-7.
Pavlides, A., and A. H. F. Chow. 2018. “Multi-objective optimization of train timetable with consideration of customer satisfaction.” Transp. Res. Rec. 2672 (8): 255–265. https://doi.org/10.1177/0361198118777629.
Qin, B., M. Yang, X. Ling, E. Chen, J. Chen, and Y. Guo. 2023. “Integrated generalized cost model considering the whole trip for urban rail transit station choices.” Transp. Res. Rec. 2678 (5): 288–304. https://doi.org/10.1177/03611981231189500.
Samà, M., P. Pellegrini, A. D’Ariano, J. Rodriguez, and D. Pacciarelli. 2016. “Ant colony optimization for the real-time train routing selection problem.” Transp. Res. Part B. Methodol. 85 (Mar): 89–108. https://doi.org/10.1016/j.trb.2016.01.005.
Serafini, P., and W. Ukovich. 1989. “A mathematical model for periodic scheduling problems.” SIAM J. Discrete Math. 2 (4): 550–581. https://doi.org/10.1137/0402049.
Sparing, D., and R. M. P. Goverde. 2017. “A cycle time optimization model for generating stable periodic railway timetables.” Transp. Res. Part B. Methodol. 98 (Apr): 198–223. https://doi.org/10.1016/j.trb.2016.12.020.
Su, B., Z. Wang, S. Su, and T. Tang. 2020. “Metro train timetable rescheduling based on q-learning approach.” In Proc., 2020 IEEE 23rd Int. Conf. on Intelligent Transportation Systems (ITSC), 1–6. New York: IEEE.
Sun, Y., and P. M. Schonfeld. 2016. “Schedule-based rail transit path-choice estimation using automatic fare collection data.” J. Transp. Eng. 142 (1): 04015037. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000812.
Sutton, R. S., and A. G. Barto. 2018. Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
Tang, R., L. De Donato, N. Besinović, F. Flammini, R. M. Goverde, Z. Lin, R. Liu, T. Tang, V. Vittorini, and Z. Wang. 2022. “A literature review of artificial intelligence applications in railway systems.” Transp. Res. Part C Emerging Technol. 140 (Jul): 103679. https://doi.org/10.1016/j.trc.2022.103679.
Tormos, P., A. Lova, F. Barber, L. Ingolotti, M. Abril, and M. A. Salido. 2008. “A genetic algorithm for railway scheduling problems.” In Metaheuristics for scheduling in industrial and manufacturing applications, 255–276. Berlin: Springer.
Wang, J., Z. Yuan, Z. Cao, and Z. Lu. 2021. “Optimal bus bridging schedule with transfer passenger demand during disruptions of urban rail transit.” J. Transp. Eng. Part A. Syst. 147 (10): 04021071. https://doi.org/10.1061/JTEPBS.0000568.
Wang, Y., and T. Tang. 2023. “A simulation-based model for evacuation demand estimation under unconventional metro emergencies.” J. Transp. Eng. Part A. Syst. 149 (7): 04023053. https://doi.org/10.1061/JTEPBS.TEENG-7682.
Yang, L., J. Qi, S. Li, and Y. Gao. 2016. “Collaborative optimization for train scheduling and train stop planning on high-speed railways.” Omega 64 (Oct): 57–76. https://doi.org/10.1016/j.omega.2015.11.003.
Yang, X., Y. Li, X. Guo, M. Ding, and J. Yang. 2023. “Simulation of energy-efficient operation for metro trains: A discrete event-driven method based on multi-agent theory.” Physica A 609 (Jan): 128325. https://doi.org/10.1016/j.physa.2022.128325.
Yin, J., T. Tang, L. Yang, J. Xun, Y. Huang, and Z. Gao. 2017. “Research and development of automatic train operation for railway transportation systems: A survey.” Transp. Res. Part C. Emerging Technol. 85 (Dec): 548–572. https://doi.org/10.1016/j.trc.2017.09.009.
Ying, C.-S., A. H. Chow, and K.-S. Chin. 2020. “An actor-critic deep reinforcement learning approach for metro train scheduling with rolling stock circulation under stochastic demand.” Transp. Res. Part B Methodol. 140 (Oct): 210–235. https://doi.org/10.1016/j.trb.2020.08.005.
Ying, C.-S., A. H. Chow, Y.-H. Wang, and K.-S. Chin. 2021. “Adaptive metro service schedule and train composition with a proximal policy optimization approach based on deep reinforcement learning.” IEEE Trans. Intell. Transp. Syst. 23 (7): 6895–6906. https://doi.org/10.1109/TITS.2021.3063399.
Zhang, T., D. Li, and Y. Qiao. 2018. “Comprehensive optimization of urban rail transit timetable by minimizing total travel times under time-dependent passenger demand and congested conditions.” Appl. Math. Modell. 58 (Jun): 421–446. https://doi.org/10.1016/j.apm.2018.02.013.
Zhou, H., J. Qi, L. Yang, J. Shi, and P. Mo. 2022. “Joint optimization of train scheduling and rolling stock circulation planning with passenger flow control on tidal overcrowded metro lines.” Transp. Res. Part C Emerging Technol. 140 (Jul): 103708. https://doi.org/10.1016/j.trc.2022.103708.
Zhu, W., W. Fan, J. Wei, and W. D. Fan. 2020. “Complete estimation approach for characterizing passenger travel time distributions at rail transit stations.” J. Transp. Eng. Part A. Syst. 146 (7): 04020050. https://doi.org/10.1061/JTEPBS.0000375.
Zhu, Z., Y. Zhang, S. Qiu, Y. Zhao, J. Ma, and Z. He. 2023. “Ridership prediction of urban rail transit stations based on AFC and POI data.” J. Transp. Eng. Part A. Syst. 149 (9): 04023077. https://doi.org/10.1061/JTEPBS.TEENG-7808.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part A: Systems
Volume 150 • Issue 9 • September 2024
Copyright
© 2024 American Society of Civil Engineers.
History
Received: Nov 14, 2023
Accepted: Apr 4, 2024
Published online: Jun 19, 2024
Published in print: Sep 1, 2024
Discussion open until: Nov 19, 2024
ASCE Technical Topics:
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.