Coordinated Ramp Metering with Equity Consideration Using Reinforcement Learning
Publication: Journal of Transportation Engineering, Part A: Systems
Volume 143, Issue 7
Abstract
Reinforcement learning (RL) has been applied to solve ramp-metering problems and attracted increasing attention in recent studies. However, improving traffic efficiency is the main concern of these applications, and the issue relating to user equity has not been well considered. A new RL-based system is developed in this paper to deal with equity-related problems. With the definition of three RL elements, including reward, action, and state, this system can capture the information of user equity and balance it with traffic efficiency. Simulation experiments using real traffic data collected from a real-world motorway stretch are designed to test the performance of the new system. Compared with a widely used ramp-metering algorithm ALINEA, the new system shows superior performance on improving both traffic efficiency and user equity. Specifically, with suitable parameter settings, the new system can reduce the total time spent (TTS) by motorway users by 18.5% and maintain an equally distributed total waiting time (TWT) with a low standard deviation for TWT across on-ramps close to 0.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
This paper is supported by China Scholarship Council and University of Leeds (CSC-University of Leeds scholarship) and partially supported by National Natural Science Foundation of China (Grant Nos. 91420203 and 51275041) and Beijing Institute of Technology Research Fund Program for Young Scholars. The authors would like to thank the institutions who support this study.
References
Bergsma, H. A. (2006). European ramp metering project (deliverable D3.2) simulation testing for local ramp metering strategies, European Commission, Brussels, Belgium.
Box, S., and Waterson, B. (2013). “An automated signalized junction controller that learns strategies by temporal difference reinforcement learning.” Eng. Appl. Artif. Intell., 26(1), 652–659.
Chow, A. H. F., and Li, Y. (2014). “Robust optimization of dynamic motorway traffic via ramp metering.” IEEE Trans. Intell. Transp. Syst., 15(3), 1374–1380.
Davarynejad, M., Hegyi, A., Vrancken, J., and van den Berg, J. (2011). “Motorway ramp-metering control with queuing consideration using Q-learning.” Proc., 14th IEEE Int. Conf. on Intelligence Transportation System (ITSC), IEEE, Washington, DC, 1652–1658.
Dervisoglu, G., Gomes, G., Kwon, J., Horowitz, R., and Varaiya, P. P. (2009). “Automatic calibration of the fundamental diagram and empirical observations on capacity.” Proc., Transportation Research Board 88th Annual Meeting, TRB, Washington, DC.
Fares, A., and Gomaa, W. (2015). “Multiagent reinforcement learning control for ramp metering.” Proc., 23rd Int. Conf. on System Engineering, Springer, Cham, Switzerland, 167–173.
Gomes, G., and Horowitz, R. (2003). “A study of two on-ramp metering schemes for congested freeways.” Proc., 2003 American Control Conf., IEEE, Washington, DC, 3756–3761.
Gomes, G., and Horowitz, R. (2006). “Optimal freeway ramp metering using the asymmetric cell transmission model.” Transp. Res. C-Emer., 14(4), 244–262.
Grant-Muller, S., and Laird, J. (2007). “The cost of congestion.” Proc., 35th European Transport Conf., Association for European Transport, Henley-in-Arden, U.K.
Haddad, J., Ramezani, M., and Geroliminis, N. (2013). “Cooperative traffic control of a mixed network with two urban regions and a freeway.” Transp. Res. B-Meth., 54, 17–36.
Han, X., and Naeher, L. P. (2006). “A review of traffic-related air pollution exposure assessment studies in the developing world.” Environ. Int., 32(1), 106–120.
Highways England. (2013). “Hatris homepage.” ⟨https://www.hatris.co.uk/⟩ (Jun. 11, 2013).
Jacob, C., and Abdulhai, B. (2006). “Automated adaptive traffic corridor control using reinforcement learning: Approach and case studies.” Transp. Res. Rec., 1959, 1–8.
Jacob, C., and Abdulhai, B. (2010). “Machine learning for multi-jurisdictional optimal traffic corridor control.” Transp. Res. A-Pol., 44(2), 53–64.
Jacobson, L. N., Henry, K. C. and Mehyar, O. (1989). “Real-time metering algorithm for centralized control.” Transp. Res. Rec., 1232, 17–26.
Kotsialos, A., and Papageorgiou, M. (2004). “Efficiency and equity properties of freeway network-wide ramp metering with AMOC.” Transp. Res. C-Emer., 12(6), 401–420.
Kotsialos, A., Papageorgiou, M., Hayden, J., Higginson, R., McCabe, K., and Rayman, N. (2006). “Discrete release rate impact on ramp metering performance.” IEE Proc. Intell. Transp. Syst., 153(1), 85–96.
Kotsialos, A., Papageorgiou, M., and Middelham, F. (2001). “Optimal coordinated ramp metering with advanced motorway optimal control.” Transp. Res. Rec., 1748, 55–65.
Lau, R. (1997). “Ramp metering by zone—The Minnesota algorithm.” Minnesota Dept. of Transportation, St. Paul, MN.
Lewis, F. L., and Vrabie, D. (2009). “Reinforcement learning and adaptive dynamic programming for feedback control.” IEEE Circuits Syst. Mag., 9(3), 32–50.
Lu, C., Chen, H., and Grant-Muller, S. (2013). “An indirect reinforcement learning approach for ramp control under incident-induced congestion.” Proc., 14th IEEE Int. Conf. on Intelligence Transportation System, IEEE, Piscataway, NJ, 979–984.
Masher, D. P., Ross, D., Wong, P., Tuan, P., Zeidler, H., and Petracek, S. (1975). Guidelines for design and operation of ramp-control systems, Stanford Research Institute, Stanford, CA.
Meng, Q., and Khoo, H. L. (2010). “A Pareto-optimization approach for a fair ramp metering.” Transp. Res. C-Emer., 18(4), 489–506.
Noland, R. B., and Quddus, M. A. (2005). “Congestion and safety: A spatial analysis of London.” Transp. Res. A: Pol. Pract., 39(7–9), 737–754.
Paesani, G., Kerr, J., Perovich, P., and Khosravi, F. (1997). “System-wide adaptive ramp metering (SWARM).” Proc., ITS America 7th Annual Meeting and Exposition: Merging the Transportation and Communications Revolutions, ITS America, Washington, DC.
Papageorgiou, M., Hadj-Salem, H., and Blosseville, J. M. (1991). “ALINEA: A local feedback control law for on-ramp metering.” Transp. Res. Rec., 1320, 58–64.
Papageorgiou, M., and Kotsialos, A. (2002). “Freeway ramp metering: an overview.” IEEE Trans. Intell. Transp. Syst., 3(4), 271–281.
Papamichail, I., Kotsialos, A., Margonis, I., and Papageorgiou, M. (2010). “Coordinated ramp metering for freeway networks—A model-predictive hierarchical control approach.” Transp. Res. C-Emer., 18(3), 311–331.
Puterman, M. L. (2009). Markov decision processes: Discrete stochastic dynamic programming, Wiley, NJ.
Rezaee, K., Abdulhai, B., and Abdelgawad, H. (2013). “Self-learning adaptive ramp metering: Analysis of design parameters on a test case in Toronto, Canada.” Transp. Res. Rec., 2396, 10–18.
Rezaee, K., Abdulhai, B., and Abdelgawad, H. (2014). “Closed-loop optimal freeway ramp metering using continuous state space reinforcement learning with function approximation.” Proc., Transportation Research Board 93rd Annual Meeting, Washington, DC.
Sutton, R. S., and Barto, A. G. (1998). Reinforcement learning: an introduction, MIT Press, Cambridge, MA.
Veljanovska, K., M., Bombol, and K., Maher, T. (2010). “Reinforcement learning technique in multiple motorway access control strategy design.” PROMET-Traffic Transp., 22(2), 117–123.
Watkins, C. C. H., and Dayan, P. (1992). “Q-learning.” Mach. Learn., 8(3–4), 279–292.
Zhang, G., and Wang, Y. (2013). “Optimizing coordinated ramp metering: A preemptive hierarchical control approach.” Comput-Aided. Civ. Inf., 28(1), 22–37.
Zhang, L., and Levinson, D. (2005). “Balancing efficiency and equity of ramp meters.” J. Transp. Eng., 477–481.
Zhao, D., Bai, X., Wang, F.-Y., Xu, J., and Yu, W. (2011). “DHP method for ramp metering of freeway traffic.” IEEE Trans. Intell. Transp. Syst., 12(4), 990–999.
Information & Authors
Information
Published In
Journal of Transportation Engineering, Part A: Systems
Volume 143 • Issue 7 • July 2017
Copyright
©2017 American Society of Civil Engineers.
History
Received: May 18, 2015
Accepted: Oct 31, 2016
Published online: Mar 20, 2017
Published in print: Jul 1, 2017
Discussion open until: Aug 20, 2017
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.