A Reinforcement Learning Method for Multiasset Roadway Improvement Scheduling Considering Traffic Impacts
Publication: Journal of Infrastructure Systems
Volume 28, Issue 4
Abstract
Maintaining roadway pavements and bridge decks is key to providing high levels of service for road users. However, improvement actions incur downtime. These actions are typically scheduled by asset class, yet implemented on any asset type, they have network-wide impacts on traffic performance. This paper presents a bilevel program wherein the upper level involves a Markov decision process (MDP) through which potential roadway improvement actions across asset classes are prioritized and scheduled. The MDP approach considers uncertainty in component deterioration effects, while incorporating the benefits of implemented improvement actions. The upper level takes as input traffic flow estimates obtained from a lower-level user equilibrium traffic formulation that recognizes changes in capacities determined by decisions taken in the upper level. Because an exact solution of this bilevel, stochastic, dynamic program is formidable, a deep reinforcement learning (DRL) method is developed. The model and solution methodology were tested on a hypothetical problem from the literature. The importance of obtaining optimal activity plans that account for downtime effects, traffic congestion impacts, uncertainty in deterioration processes, and multiasset classes is demonstrated.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data used in the study are provided in the paper and/or Supplemental Materials.
Acknowledgments
This work was sponsored by a grant from the Center for Integrated Asset Management for Multimodal Transportation Infrastructure Systems (CIAMTIS), a USDOT University Transportation Center, under Federal Grant No. 69A3551847103. The authors are grateful for the support.
References
Andriotis, C. P., and K. G. Papakonstantinou. 2019. “Managing engineering systems with large state and action spaces through deep reinforcement learning.” Reliab. Eng. Syst. Saf. 191 (Nov): 106483. https://doi.org/10.1016/j.ress.2019.04.036.
Andriotis, C. P., and K. G. Papakonstantinou. 2021. “Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints.” Reliab. Eng. Syst. Saf. 212 (Aug): 107551. https://doi.org/10.1016/j.ress.2021.107551.
Beckmann, M., C. McGuire, and C. Winsten. 1956. Studies in the economics of transportation. New Haven, CT: Yale University Press.
Chatti, K., and I. Zaabar. 2012. Estimating the effects of pavement condition on vehicle operating costs. Washington, DC: Transportation Research Board.
Chu, J. C., and Y.-J. Chen. 2012. “Optimal threshold-based network-level transportation infrastructure life-cycle management with heterogeneous maintenance actions.” Transp. Res. Part B Methodol. 46 (9): 1123–1143. https://doi.org/10.1016/j.trb.2012.05.002.
Chu, J. C., and K.-H. Huang. 2018. “Mathematical programming framework for modeling and comparing network-level pavement maintenance strategies.” Transp. Res. Part B Methodol. 109 (Mar): 1–25. https://doi.org/10.1016/j.trb.2018.01.005.
Durango-Cohen, P. L. 2004. “Maintenance and repair decision making for infrastructure facilities without a deterioration model.” J. Infrastruct. Syst. 10 (1): 1–8. https://doi.org/10.1061/(ASCE)1076-0342(2004)10:1(1).
Durango-Cohen, P. L., and P. Sarutipand. 2009. “Maintenance optimization for transportation systems with demand responsiveness.” Transp. Res. Part C Emerging Technol. 17 (4): 337–348. https://doi.org/10.1016/j.trc.2009.01.001.
Faddoul, R., W. Raphael, A.-H. Soubra, and A. Chateauneuf. 2013. “Incorporating Bayesian networks in Markov decision processes.” J. Infrastruct. Syst. 19 (4): 415–424. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000134.
George, K. P., A. S. Rajagopal, and L. K. Lim. 1989. “Models for predicting pavement deterioration.” Transp. Res. Rec. (1215): 1–7.
Konda, V., and J. Tsitsiklis. 2003. “On actor-critic algorithms.” SIAM J. Control Optim. 42 (4): 1143–1166. https://doi.org/10.1137/S0363012901385691.
Labi, S., and K. C. Sinha. 2003. “Measures of short-term effectiveness of highway pavement maintenance.” J. Transp. Eng. 129 (6): 673–683. https://doi.org/10.1061/(ASCE)0733-947X(2003)129:6(673).
Li, Y., and S. Madanat. 2002. “A steady-state solution for the optimal pavement resurfacing problem.” Transp. Res. Part A Policy Pract. 36 (6): 525–535. https://doi.org/10.1016/S0965-8564(01)00020-9.
Madanat, S., and M. Ben-Akiva. 1994. “Optimal inspection and repair policies for infrastructure facilities.” Transp. Sci. 28 (1): 55–62. https://doi.org/10.1287/trsc.28.1.55.
Manafpour, A., I. Guler, A. Radlińska, F. Rajabipour, and G. Warn. 2018. “Stochastic analysis and time-based modeling of concrete bridge deck deterioration.” J. Bridge Eng. 23 (9): 04018066. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001285.
Medury, A., and S. Madanat. 2013. “Incorporating network considerations into pavement management systems: A case for approximate dynamic programming.” Transp. Res. Part C Emerging Technol. 33 (Aug): 134–150. https://doi.org/10.1016/j.trc.2013.03.003.
Mnih, V., et al. 2015. “Human-level control through deep reinforcement learning.” Nature 518 (7540): 529–533. https://doi.org/10.1038/nature14236.
Mnih, V., A. P. Badia, M. Mirza, A. Graves, T. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. 2016. “Asynchronous methods for deep reinforcement learning.” In Proc., Int. Conf. on Machine Learning, 1928–1937. Ithaca, NY: Cornell Univ.
Ng, M., D. Y. Lin, and S. T. Waller. 2009. “Optimal long-term infrastructure maintenance planning accounting for traffic dynamics.” Comput.-Aided Civ. Infrastruct. Eng. 24 (7): 459–469. https://doi.org/10.1111/j.1467-8667.2009.00606.x.
Ouyang, Y. 2007. “Pavement resurfacing planning for highway networks: Parametric policy iteration approach.” J. Infrastruct. Syst. 13 (1): 65–71. https://doi.org/10.1061/(ASCE)1076-0342(2007)13:1(65).
Ouyang, Y., and S. Madanat. 2004. “Optimal scheduling of rehabilitation activities for multiple pavement facilities: Exact and approximate solutions.” Transp. Res. Part A Policy Pract. 38 (5): 347–365. https://doi.org/10.1016/j.tra.2003.10.007.
Ouyang, Y., and S. Madanat. 2006. “An analytical solution for the finite-horizon pavement resurfacing planning problem.” Transp. Res. Part B Methodol. 40 (9): 767–778. https://doi.org/10.1016/j.trb.2005.11.001.
Papakonstantinou, K. G., and M. Shinozuka. 2014. “Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part II: POMDP implementation.” Reliab. Eng. Syst. Saf. 130 (Oct): 214–224. https://doi.org/10.1016/j.ress.2014.04.006.
PennDOT (Pennsylvania Department of Transportation) and Bureau of Design. 2009. Bridge management system 2 (BMS2) coding manual: Office version. Harrisburg, PA: PennDOT.
Puterman, M. 1994. Markov decision processes: Discrete stochastic dynamic programming. Hoboken, NJ: Wiley.
Sutton, R., and A. Barto. 2018. Reinforcement learning, second edition: An introduction. Cambridge, MA: MIT Press.
Sutton, R., H. Maei, D. Precup, S. Bhatnagar, D. Silver, C. Szepesvári, and E. Wiewiora. 2009. “Fast gradient-descent methods for temporal-difference learning with linear function approximation.” In Proc., 26th Annual Int. Conf. on Machine Learning, ICML ’09, 993–1000. New York: Association for Computing Machinery.
US Department of Commerce. 1964. Traffic assignment manual for application with a large, high speed computer. Washington, DC: US Dept. of Commerce, Bureau of Public Roads, Office of Planning, Urban Planning Division.
USDOT. 2016. The value of travel time savings: Departmental guidance for conducting economic evaluations. Washington, DC: USDOT.
USDOT. 2021. “Simplified highway capacity calculation method for the highway performance monitoring system.” Accessed February 15, 2022. https://www.fhwa.dot.gov/policyinformation/pubs/pl18003/chap031067.cfm. 27106.
USDOT, FHWA (Federal Highway Administration). 2017. National household travel survey. Washington, DC: USDOT, FHWA.
VDOT (Virginia DOT). 2016. VDOT: Supporting document for the development and enhancement of the pavement maintenance decision matrices used in the needs-based analysis. Richmond, VA: VDOT.
Walls, J., and M. R. Smith. 1998. Life-cycle cost analysis in pavement design: Interim technical bulletin. Washington, DC: USDOT, Federal Highway Administration.
Wang, T., J. Harvey, J. Lea, and C. Kim. 2014. “Impact of pavement roughness on vehicle free-flow speed.” J. Transp. Eng. 140 (9): 04014039. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000689.
Wei, C., and S. Tighe. 2004. “Development of preventive maintenance decision trees based on cost-effectiveness analysis: An Ontario case study.” Transp. Res. Rec. 1866 (1): 9–19. https://doi.org/10.3141/1866-02.
Werbos, P. J. 1988. “Generalization of backpropagation with application to a recurrent gas market model.” Neural Networks 1 (4): 339–356. https://doi.org/10.1016/0893-6080(88)90007-X.
Information & Authors
Information
Published In
Journal of Infrastructure Systems
Volume 28 • Issue 4 • December 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Sep 30, 2021
Accepted: Apr 15, 2022
Published online: Sep 7, 2022
Published in print: Dec 1, 2022
Discussion open until: Feb 7, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Meet Saiya, Aditya Medury, Incorporating Economies of Scale in Top-Down Pavement Management Systems, Journal of Infrastructure Systems, 10.1061/JITSE4.ISENG-2527, 30, 4, (2024).
- Caio Filipe de Lima Munguba, Gustavo de Novaes Pires Leite, Alvaro Antonio Villa Ochoa, Enrique Lopez Droguett, Condition-based maintenance with reinforcement learning for refrigeration systems with selected monitored features, Engineering Applications of Artificial Intelligence, 10.1016/j.engappai.2023.106067, 122, (106067), (2023).
- Sohrab Mamdoohi, Elise Miller-Hooks, Machine Learning and Reverse Methods for a Deeper Understanding of Public Roadway Improvement Action Impacts during Execution, Journal of Advanced Transportation, 10.1155/2022/6385236, 2022, (1-22), (2022).