Deep Reinforcement Learning–Enabled Bridge Management Considering Asset and Network Risks
Publication: Journal of Infrastructure Systems
Volume 28, Issue 3
Abstract
Bridges deteriorate over time due to various environmental and mechanical stressors. Deterioration is a significant risk to bridge owners (asset risk) and the traveling public (network risk). To tackle this issue, transportation agencies carry out bridge management under limited resources to preserve bridge conditions and control the risks of bridge failure. Nonetheless, existing network-level analysis for bridge management cannot explicitly consider the effects of preservation actions on network risk, measured directly by functionality indicators such as network capacity. In this paper, a novel method based on deep reinforcement learning is proposed to devise network-level preservation policies that can reflect bridge importance to network functionality. The proposed method is based on the proximal policy optimization algorithm adapted for bridge management problems and improved via distributed computing and architecture. The method is applied to an illustrative bridge network. The results indicate that the proposed method can produce significantly better preservation policies in terms of minimizing long-term costs that include asset and network risks. The devised policies are also investigated in depth to allow for transparent interpretation and easy integration with existing bridge management systems.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author on reasonable request. Specifically, the trained actor and critic networks used in the illustrative example are available on request.
Acknowledgments
The author is grateful for the financial support received from Portland State University. The opinions and conclusions presented in this paper are those of the author and do not necessarily reflect the views of the sponsoring organization.
References
AASHTO. 2018. AASHTO transportation asset management guide. Washington, DC: AASHTO.
AASHTO. 2019. AASHTO manual for bridge element inspection. Washington, DC: AASHTO.
AASHTO. 2021. “AASHTOWare bridge management.” Accessed November 20, 2021. https://www.aashtowarebridge.com.
Abadi, M., et al. 2015. “TensorFlow: Large-scale machine learning on heterogeneous systems.” Accessed December 31, 2019. https://www.tensorflow.org/.
Adey, B. T. 2019. “A road infrastructure asset management process: Gains in efficiency and effectiveness.” Infrastruct. Asset Manage. 6 (1): 2–14. https://doi.org/10.1680/jinam.17.00018.
Andriotis, C., and K. Papakonstantinou. 2021. “Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints.” Reliab. Eng. Syst. Saf. 212 (20): 107551. https://doi.org/10.1016/j.ress.2021.107551.
Andriotis, C., and K. G. Papakonstantinou. 2019. “Managing engineering systems with large state and action spaces through deep reinforcement learning.” Reliab. Eng. Syst. Saf. 191 (Apr): 106483. https://doi.org/10.1016/j.ress.2019.04.036.
Assaad, R., and I. H. El-adaway. 2020. “Bridge infrastructure asset management system: Comparative computational machine learning approach for evaluating and predicting deck deterioration conditions.” J. Infrastruct. Syst. 26 (3): 04020032. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000572.
Bishop, C. M. 2006. Pattern recognition and machine learning. New York: Springer.
Burkhalter, M., and B. T. Adey. 2020. “Modelling the complex relationship between interventions, interventions costs and the service provided when evaluating intervention programs on railway infrastructure networks.” Infrastructures 5 (12): 113. https://doi.org/10.3390/infrastructures5120113.
Chang, L., F. Peng, Y. Ouyang, A. S. Elnashai, and B. F. Spencer. 2012. “Bridge seismic retrofit program planning to maximize postearthquake transportation network capacity.” J. Infrastruct. Syst. 18 (2): 75–88. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000082.
Chang, M., M. Maguire, and Y. Sun. 2019. “Stochastic modeling of bridge deterioration using classification tree and logistic regression.” J. Infrastruct. Syst. 25 (1): 04018041. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000466.
Dehghani, N. L., A. B. Jeddi, and A. Shafieezadeh. 2021. “Intelligent hurricane resilience enhancement of power distribution systems via deep reinforcement learning.” Appl. Energy 285 (22): 116355. https://doi.org/10.1016/j.apenergy.2020.116355.
Fecarotti, C., J. Andrews, and R. Pesenti. 2021. “A mathematical programming model to select maintenance strategies in railway networks.” Reliab. Eng. Syst. Saf. 216 (Jan): 107940. https://doi.org/10.1016/j.ress.2021.107940.
Fernando, D., B. T. Adey, and N. Lethanh. 2015. “A model for the evaluation of intervention strategies for bridges affected by manifest and latent deterioration processes.” Struct. Infrastruct. Eng. 11 (11): 1466–1483. https://doi.org/10.1080/15732479.2014.976576.
Fernando, D., B. T. Adey, and S. Walbridge. 2013. “A methodology for the prediction of structure level costs based on element condition states.” Struct. Infrastruct. Eng. 9 (8): 735–748. https://doi.org/10.1080/15732479.2011.609176.
FHWA (Federal Highway Administration). 2012. Bridge inspector’s reference manual. Washington, DC: FHWA.
FHWA (Federal Highway Administration). 2021. “FHWA resources for the asset management practitioner.” Accessed November 5, 2021. https://www.fhwa.dot.gov/asset/pubs/asset_management_practitioner.pdf.
Ford, L. R., and D. R. Fulkerson. 1956. “Maximal flow through a network.” Can. J. Math. 8 (6): 399–404. https://doi.org/10.4153/CJM-1956-045-5.
Frangopol, D. M. 2011. “Life-cycle performance, management, and optimisation of structural systems under uncertainty: Accomplishments and challenges.” Struct. Infrastruct. Eng. 7 (6): 389–413. https://doi.org/10.1080/15732471003594427.
Frangopol, D. M., K.-Y. Lin, and A. C. Estes. 1997. “Life-cycle cost design of deteriorating structures.” J. Struct. Eng. 123 (10): 1390–1401. https://doi.org/10.1061/(ASCE)0733-9445(1997)123:10(1390).
Furuya, A., and S. Madanat. 2013. “Accounting for network effects in railway asset management.” J. Transp. Eng. 139 (1): 92–100. https://doi.org/10.1061/(ASCE)TE.1943-5436.0000477.
Golabi, K., R. B. Kulkarni, and G. B. Way. 1982. “A statewide pavement management system.” Interfaces 12 (6): 5–21. https://doi.org/10.1287/inte.12.6.5.
Golabi, K., and R. Shepard. 1997. “Pontis: A system for maintenance optimization and improvement of US bridge networks.” Interfaces 27 (1): 71–88. https://doi.org/10.1287/inte.27.1.71.
Goyal, R., M. J. Whelan, and T. L. Cavalline. 2020. “Multivariable proportional hazards-based probabilistic model for bridge deterioration forecasting.” J. Infrastruct. Syst. 26 (2): 04020007. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000534.
Graesser, L., and W. L. Keng. 2020. Foundations of deep reinforcement learning. Boston, MA: Addison-Wesley.
Guadarrama, S., et al. 2018. “TF-Agents: A library for reinforcement learning in tensorflow.” Accessed November 24, 2021. https://github.com/tensorflow/agents.
Hagberg, A., P. Swart, and S. Chult. 2008. “Exploring network structure, dynamics, and function using network.” In Proc., 7th Python in Science Conf. (SciPy2008). Washington, DC: US Department of Energy.
Idaho Transportation Department. 2021. Idaho manual for bridge evaluation. Boise, ID: Idaho Transportation Department.
Lee, J., H. Guan, Y.-C. Loo, and M. Blumenstein. 2014. “Development of a long-term bridge element performance model using Elman neural networks.” J. Infrastruct. Syst. 20 (3): 04014013. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000197.
Lethanh, N., B. T. Adey, and M. Burkhalter. 2018. “Determining an optimal set of work zones on large infrastructure networks in a GIS framework.” J. Infrastruct. Syst. 24 (1): 04017048. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000410.
Lethanh, N., J. Hackl, and B. T. Adey. 2017. “Determination of Markov transition probabilities to be used in bridge management from mechanistic-empirical models.” J. Bridge Eng. 22 (10): 04017063. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001101.
Liu, H., and Y. Zhang. 2020. “Bridge condition rating data modeling using deep learning algorithm.” Struct. Infrastruct. Eng. 16 (10): 1447–1460. https://doi.org/10.1080/15732479.2020.1712610.
Liu, L., D. Y. Yang, and D. M. Frangopol. 2021. “Determining target reliability index of structures based on cost optimization and acceptance criteria for fatality risk.” J. Risk Uncertainty Eng. Syst. Part A: Civ. Eng. 7 (2): 04021013. https://doi.org/10.1061/AJRUA6.0001127.
Medury, A., and S. Madanat. 2013. “Incorporating network considerations into pavement management systems: A case for approximate dynamic programming.” Transp. Res. Part C Emerging Technol. 33 (Mar): 134–150. https://doi.org/10.1016/j.trc.2013.03.003.
Mirzaei, Z., and B. T. Adey. 2015. “Investigation of the use of three existing methodologies to determine optimal life-cycle activity profiles for bridges.” Struct. Infrastruct. Eng. 11 (11): 1484–1509. https://doi.org/10.1080/15732479.2014.976577.
ODOT (Oregon DOT). 2003. Economic and bridge options report. Salem, OR: ODOT.
ODOT (Oregon DOT). 2020. ODOT’s bridge condition report and tunnel data. Salem, OR: ODOT.
Patidar, V., S. Labi, K. C. Sinha, and P. Thompson. 2007. NCHRP report 590: Multi-objective optimization for bridge management systems. Washington, DC: Transportation Research Board.
Patriksson, M. 2015. The traffic assignment problem: Models and methods. New York: Dover Publications.
Ruiz, F., A. Aguado, C. Serrat, and J. R. Casas. 2019. “Optimal metric for condition rating of existing buildings: Is five the right number?” Struct. Infrastruct. Eng. 15 (6): 740–753. https://doi.org/10.1080/15732479.2018.1557702.
Saydam, D., D. M. Frangopol, and Y. Dong. 2013. “Assessment of risk using bridge element condition ratings.” J. Infrastruct. Syst. 19 (3): 252–265. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000131.
Schulman, J., S. Levine, P. Abbeel, M. Jordan, and P. Moritz. 2015. “Trust region policy optimization.” In Proc., 32nd Int. Conf. on Machine Learning. Brookline, MA: Microtome Publishing.
Schulman, J., F. Wolski, P. Dhariwal, A. Radford, and O. Klimov. 2017. “Proximal policy optimization algorithms.” Preprint, submitted July 20, 2017. https://arXiv:1707.06347.
Sobanjo, J. O. 2011. “State transition probabilities in bridge deterioration based on Weibull sojourn times.” Struct. Infrastruct. Eng. 7 (10): 747–764. https://doi.org/10.1080/15732470902917028.
Sobanjo, J. O., and P. D. Thompson. 2011. Enhancement of the FDOT’s project level and network level bridge management analysis tools. Tallahassee, FL: Florida DOT.
Sutton, R. S., and A. G. Barto. 2018. Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
Thomas, O., and J. Sobanjo. 2016. “Semi-Markov models for the deterioration of bridge elements.” J. Infrastruct. Syst. 22 (3): 04016010. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000298.
Thompson, P. D., E. P. Small, M. Johnson, and A. R. Marshall. 1998. “The Pontis bridge management system.” Struct. Eng. Int. 8 (4): 303–308. https://doi.org/10.2749/101686698780488758.
Yamany, M. S., D. M. Abraham, and S. Labi. 2021. “Comparative analysis of Markovian methodologies for modeling infrastructure system performance.” J. Infrastruct. Syst. 27 (2): 04021003. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000604.
Yang, D. Y. 2022. “Adaptive risk-based life-cycle management for large-scale structures using deep reinforcement learning and surrogate modeling.” J. Eng. Mech. 148 (1): 1–15. https://doi.org/10.1061/(ASCE)EM.1943-7889.0002028.
Yang, D. Y., and D. M. Frangopol. 2018. “Risk-informed bridge ranking at project and network levels.” J. Infrastruct. Syst. 24 (3): 04018018. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000430.
Yang, D. Y., and D. M. Frangopol. 2020. “Life-cycle management of deteriorating bridge networks with network-level risk bounds and system reliability analysis.” Struct. Saf. 83 (Mar): 101911. https://doi.org/10.1016/j.strusafe.2019.101911.
Yang, D. Y., and D. M. Frangopol. 2021. “Risk-based inspection planning of deteriorating structures.” Struct. Infrastruct. Eng. 2021 (Mar): 1–12. https://doi.org/10.1080/15732479.2021.1994617.
Information & Authors
Information
Published In
Journal of Infrastructure Systems
Volume 28 • Issue 3 • September 2022
Copyright
© 2022 American Society of Civil Engineers.
History
Received: Dec 8, 2021
Accepted: May 5, 2022
Published online: Jul 7, 2022
Published in print: Sep 1, 2022
Discussion open until: Dec 7, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Zachary Hamida, James-A. Goulet, Quantifying the Relative Change in Maintenance Costs due to Delayed Maintenance Actions in Transportation Infrastructure, Journal of Performance of Constructed Facilities, 10.1061/JPCFEV.CFENG-4802, 38, 5, (2024).
- Zachary Hamida, James-A. Goulet, Hierarchical reinforcement learning for transportation infrastructure maintenance planning, Reliability Engineering & System Safety, 10.1016/j.ress.2023.109214, 235, (109214), (2023).