A Decision-Making Framework for Load Rating Planning of Aging Bridges Using Deep Reinforcement Learning
Publication: Journal of Computing in Civil Engineering
Volume 35, Issue 6
Abstract
Load rating is gaining popularity as a method for inspecting the structural performance of aging bridges and determining maintenance actions. Cost-effective condition-based strategies have been developed in previous studies to balance the additional costs and structural safety. However, due to the lack of constant replacement thresholds usually stipulated in governmental guidelines, they may not be suitable in practice. Furthermore, those studies neglected the preferences of decision makers, which influences the choice of optimal plans. This paper proposes a decision-making framework incorporating risk attitudes and time preference for a cost-effective load rating strategy. This strategy utilizes replacement thresholds from current guidelines and determines the time of the next load rating adaptively based on the observation results. It is formulated as a Markov decision process (MDP) compatible with discounted utility theory. Deep reinforcement learning (DRL) is employed to solve the MDP efficiently for a bridge system with large state space. Special focus is given to hyperbolic discounting, one popular type of time preference. Its inconsistency with the MDP formulation is addressed by DRL implemented with auxiliary tasks that simultaneously learns multiple functions. An existing multigirder bridge was used as an illustrative example. Results showed that DRL can obtain cost-efficient load rating plans tailored to preferences of decision makers.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some data, models, or code that support the findings of this study are available from the first author upon reasonable request. Specifically, the source code associated with deep reinforcement learning is available.
Acknowledgments
The authors are grateful for the financial support received from the US Department of Transportation Region 3 University Transportation Center (CIAMTIS—The Center for Integrated Asset Management for Multimodal Transportation Infrastructure Systems), the US Office of Naval Research (ONR) Award N00014-16-1-2299, the US National Science Foundation Grant CMMI-1537926, and the Pennsylvania Infrastructure Technology Alliance (PITA). The opinions and conclusions presented in this paper are those of the authors and do not necessarily reflect the views of the sponsoring organizations.
References
AASHTO. 2017. AASHTO LRFD bridge design specifications. Washington, DC: AASHTO.
AASHTO. 2018. The manual for bridge evaluation. Washington, DC: AASHTO.
Akgül, F., and D. M. Frangopol. 2003. “Rating and reliability of existing bridges in a network.” J. Bridge Eng. 8 (6): 383–392. https://doi.org/10.1061/(ASCE)1084-0702(2003)8:6(383).
Akgül, F., and D. M. Frangopol. 2004. “Computational platform for predicting lifetime system reliability profiles for different structure types in a network.” J. Comput. Civ. Eng. 18 (2): 92–104. https://doi.org/10.1061/(ASCE)0887-3801(2004)18:2(92).
Alampalli, S., D. M. Frangopol, J. Grimson, M. W. Halling, D. E. Kosnik, E. O. Lantsoght, D. Yang, and Y. E. Zhou. 2021. “Bridge load testing: State-of-the-practice.” J. Bridge Eng. 26 (3): 03120002. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001678.
Albrecht, P., and A. H. Naeemi. 1984. Performance of weathering steel in bridges. Washington, DC: Transportation Research Board, National Research Council.
Alipour, M., D. K. Harris, L. E. Barnes, O. E. Ozbulut, and J. Carroll. 2017. “Load-capacity rating of bridge populations through machine learning: Application of decision trees and random forests.” J. Bridge Eng. 22 (10): 04017076. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001103.
Andriotis, C. P., and K. G. Papakonstantinou. 2019. “Managing engineering systems with large state and action spaces through deep reinforcement learning.” Reliab. Eng. Syst. Saf. 191 (Nov): 106483. https://doi.org/10.1016/j.ress.2019.04.036.
Andriotis, C. P., and K. G. Papakonstantinou. 2020. “Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints.” Preprint, submitted July 2, 2015. http://arxiv.org/abs/2007.01380.
Arulkumaran, K., M. P. Deisenroth, M. Brundage, and A. A. Bharath. 2017. “Deep reinforcement learning: A brief survey.” IEEE Signal Process. Mag. 34 (6): 26–38. https://doi.org/10.1109/MSP.2017.2743240.
Biondini, F., and D. M. Frangopol. 2018. “Life-cycle performance of civil structure and infrastructure systems: Survey.” J. Struct. Eng. 144 (1): 06017008. https://doi.org/10.1061/(ASCE)ST.1943-541X.0001923.
Borkar, V. S. 2002. “Q-learning for risk-sensitive control.” Math. Oper. Res. 27 (2): 294–311. https://doi.org/10.1287/moor.27.2.294.324.
Bucher, C., and D. M. Frangopol. 2006. “Optimization of lifetime maintenance strategies for deteriorating structures considering probabilities of violating safety, condition, and cost thresholds.” Probab. Eng. Mech. 21 (1): 1–8. https://doi:10.1016/j.probengmech.2005.06.002.
Castanier, B., A. Grall, and C. Bérenguer. 2005. “A condition-based maintenance policy with non-periodic inspections for a two-unit series system.” Reliab. Eng. Syst. Saf. 87 (1): 109–120. https://doi.org/10.1016/j.ress.2004.04.013.
Cha, E. J., and B. R. Ellingwood. 2012. “Risk-averse decision-making for civil infrastructure exposed to low-probability, high-consequence events.” Reliab. Eng. Syst. Saf. 104 (1): 27–35. https://doi.org/10.1016/j.ress.2012.04.002.
Cha, E. J., and B. R. Ellingwood. 2013. “Seismic risk mitigation of building structures: The role of risk aversion.” Struct. Saf. 40: 11–19. https://doi.org/10.1016/j.strusafe.2012.06.004.
Cha, E. J., and B. R. Ellingwood. 2014. “Attitudes towards acceptance of risk to buildings from extreme winds.” Struct. Infrastruct. Eng. 10 (6): 697–707. https://doi.org/10.1080/15732479.2012.758642.
Cheng, M., and D. M. Frangopol. 2021. “Optimal load rating-based inspection planning of corroded steel girders using Markov decision process.” Probab. Eng. Mech. https://doi.org/10.1016/j.probengmech.2021.103160.
Cheng, M., D. M. Frangopol, and C. Gong. 2020a. “Acceptance of corrosion-resistant steel in design of steel girder bridges based on expected utility theory.” J. Bridge Eng. 25 (11): 04020098. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001627.
Cheng, M., D. Y. Yang, and D. M. Frangopol. 2020b. “Investigation of effects of time preference and risk perception on life-cycle management of civil infrastructure.” ASCE-ASME J. Risk Uncertainty Eng. Syst. Part A: Civ. Eng. 6 (1): 04020001. https://doi.org/10.1061/AJRUA6.0001039.
Estes, A. C., and D. M. Frangopol. 1999. “Repair optimization of highway bridges using system reliability approach.” J. Struct. Eng. 125 (7): 766–775. https://doi.org/10.1061/(ASCE)0733-9445(1999)125:7(766).
FDOT (Florida DOT). 2016. Bridge load rating manual. Tallahassee, FL: FDOT.
Fedus, W., C. Gelada, Y. Bengio, M. G. Bellemare, and H. Larochelle. 2019. “Hyperbolic discounting and learning over multiple horizons.” Preprint, submitted February 19, 2019. http://arxiv.org/abs/1902.06865.
FHWA (Federal Highway Administration). 2015. Steel bridge design handbook: Load rating of steel bridges. Washington, DC: FHWA.
Frangopol, D. M., Y. Dong, and S. Sabatino. 2017. “Bridge life-cycle performance and cost: Analysis, prediction, optimisation and decision-making.” Struct. Infrastruct. Eng. 13 (10): 1239–1257. https://doi.org/10.1080/15732479.2016.1267772.
Frederick, S., G. Loewenstein, and T. O’Donoghue. 2002. “Time discounting and time preference: A critical review.” J. Econ. Lit. 40 (2): 351–401. https://doi.org/10.1257/jel.40.2.351.
Garcıa, J., and F. Fernández. 2015. “A comprehensive survey on safe reinforcement learning.” J. Mach. Learn. Res. 16 (1): 1437–1480.
Glorot, X., A. Bordes, and Y. Bengio. 2011. “Deep sparse rectifier neural networks.” In Proc., 14th Int. Conf. on Artificial Intelligence and Statistics. Fort Lauderdale, FL: Proceedings of Machine Learning Research.
Goda, K., and H. P. Hong. 2006. “Optimal seismic design considering risk attitude, societal tolerable risk level, and life quality criterion.” J. Struct. Eng. 132 (12): 2027–2035. https://doi.org/10.1061/(ASCE)0733-9445(2006)132:12(2027).
Gong, C., and D. M. Frangopol. 2019. “Preferred dry-docking interval of corroded ship hull girders based on cumulative prospect theory.” Ocean Eng. 192 (Nov): 106440. https://doi.org/10.1016/j.oceaneng.2019.106440.
Gong, C., and D. M. Frangopol. 2020a. “Condition-based multi-objective maintenance decision-making for highway bridges considering risk perceptions.” J. Struct. Eng. 146 (5): 04020051. https://doi.org/10.1061/(ASCE)ST.1943-541X.0002570.
Gong, C., and D. M. Frangopol. 2020b. “Reliability of steel girder bridges with dependent corrosion growth.” Eng. Struct. 224 (Dec): 111125. https://doi.org/10.1016/j.engstruct.2020.111125.
Grall, A., C. Bérenguer, and L. Dieulle. 2002. “A condition-based maintenance policy for stochastically deteriorating systems.” Reliab. Eng. Syst. Saf. 76 (2): 167–180. https://doi.org/10.1016/S0951-8320(01)00148-X.
Heymsfield, E., S. V. Hernandez, and K. Pasley. 2020. “Bridge load posting based on actual Arkansas truck traffic.” J. Bridge Eng. 25 (5): 04020018. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001539.
Howard, R. A., and J. E. Matheson. 1972. “Risk-sensitive Markov decision processes.” Manage. Sci. 18 (7): 356–369. https://doi.org/10.1287/mnsc.18.7.356.
Jaderberg, M., V. Mnih, W. M. Czarnecki, T. Schaul, J. Z. Leibo, D. Silver, and K. Kavukcuoglu. 2016. “Reinforcement learning with unsupervised auxiliary tasks.” Preprint, submitted November 16, 2016. http://arxiv.org/abs/1611.05397.
Kingma, D. P., and J. Ba. 2014. “Adam: A method for stochastic optimization.” Preprint, submitted December 22, 2014. http://arxiv.org/abs/1412.6980.
Koopmans, T. C. 1960. “Stationary ordinal utility and impatience.” Econometrica 28 (2): 287–309. https://doi.org/10.2307/1907722.
Lee, J. Y., and B. R. Ellingwood. 2015. “Ethical discounting for civil infrastructure decisions extending over multiple generations.” Struct. Saf. 57 (Nov): 43–52. https://doi.org/10.1016/j.strusafe.2015.06.001.
Li, Y. 2017. “Deep reinforcement learning: An overview.” Preprint, submitted January 25, 2017. http://arxiv.org/abs/1701.07274.
Liu, Y., Y. Chen, and T. Jiang. 2020. “Dynamic selective maintenance optimization for multi-state systems over a finite horizon: A deep reinforcement learning approach.” Eur. J. Oper. Res. 283 (1): 166–181. https://doi.org/10.1016/j.ejor.2019.10.049.
Maine DOT. 2015. Load rating guide. Augusta, ME: Maine DOT.
Mihatsch, O., and R. Neuneier. 2002. “Risk-sensitive reinforcement learning.” Mach. Learn. 49 (2): 267–290. https://doi.org/10.1023/A:1017940631555.
Mnih, V., et al. 2015. “Human-level control through deep reinforcement learning.” Nature 518 (7540): 529–533. https://doi.org/10.1038/nature14236.
Nowak, A. S. 1999. Calibration of LRFD bridge design code. Washington, DC: Transportation Research Board, National Research Council.
Okasha, N. M., and D. M. Frangopol. 2010. “Novel approach for multi-criteria optimization of life-cycle preventive and essential maintenance of deteriorating structures.” J. Struct. Eng. 136 (8): 1009–1022. https://doi.org/10.1061/(ASCE)ST.1943-541X.0000198.
Padgett, J. E., K. Dennemann, and J. Ghosh. 2010. “Risk-based seismic life-cycle cost–benefit (LCC-B) analysis for bridge retrofit assessment.” Struct. Saf. 32 (3): 165–173. https://doi.org/10.1016/j.strusafe.2009.10.003.
Papakonstantinou, K. G., and M. Shinozuka. 2014a. “Optimum inspection and maintenance policies for corroded structures using partially observable Markov decision processes and stochastic, physically based models.” Probab. Eng. Mech. 37 (Jul): 93–108. https://doi.org/10.1016/j.probengmech.2014.06.002.
Papakonstantinou, K. G., and M. Shinozuka. 2014b. “Planning structural inspection and maintenance policies via dynamic programming and Markov processes. Part II: POMDP implementation.” Reliab. Eng. Syst. Saf. 130 (Oct): 214–224. https://doi.org/10.1016/j.ress.2014.04.006.
Paszke, A., S. Gross, S. Chintala, G. Chanan, E. Yang, Z. DeVito, Z. Lin, A. Desmaison, L. Antiga, and A. Lerer. 2017. “Automatic differentiation in PyTorch.” In Proc., 31st Conf. on Neural Information Processing Systems, 1–4. San Diego: Neural Information Processing Systems Foundation.
Powell, W. B. 2007. Approximate dynamic programming: Solving the curses of dimensionality. Hoboken, NJ: Wiley.
Prashanth, L. A., and M. Fu. 2018. “Risk-sensitive reinforcement learning: A constrained optimization viewpoint.” Preprint, submitted October 22, 2018. http://arxiv.org/abs/1810.09126.
Sabatino, S., D. M. Frangopol, and Y. Dong. 2015. “Sustainability-informed maintenance optimization of highway bridges considering multi-attribute utility and risk attitude.” Eng. Struct. 102 (Nov): 310–321. https://doi.org/10.1016/j.engstruct.2015.07.030.
Samuelson, P. A. 1937. “A note on measurement of utility.” Rev. Econ. Stud. 4 (2): 155–161. https://doi.org/10.2307/2967612.
Shen, Y., M. J. Tobia, T. Sommer, and K. Obermayer. 2014. “Risk-sensitive reinforcement learning.” Neural Comput. 26 (7): 1298–1328. https://doi.org/10.1162/NECO_a_00600.
Silver, D., et al. 2016. “Mastering the game of Go with deep neural networks and tree search.” Nature 529 (7587): 484–489. https://doi.org/10.1038/nature16961.
Sozou, P. D. 1998. “On hyperbolic discounting and uncertain hazard rates.” Proc. R. Soc. London, Ser. B: Biol. Sci. 265 (1409): 2015–2020. https://doi.org/10.1098/rspb.1998.0534.
Stein, S. M., G. K. Young, R. E. Trent, and D. R. Pearson. 1999. “Prioritizing scour vulnerable bridges using risk.” J. Infrastruct. Syst. 5 (3): 95–101. https://doi.org/10.1061/(ASCE)1076-0342(1999)5:3(95).
Sutton, R. S., and A. G. Barto. 2018. Reinforcement learning: An introduction. Cambridge, MA: MIT Press.
Wang, N., C. O’Malley, B. R. Ellingwood, and A.-H. Zureick. 2011. “Bridge rating using system reliability assessment. I: Assessment and verification by load testing.” J. Bridge Eng. 16 (6): 854–862. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000172.
Watkins, C. J., and P. Dayan. 1992. “Q-learning.” Mach. Learn. 8 (3–4): 279–292. https://doi.org/10.1007%2F978-1-4615-3618-5_4.
Wei, S., Y. Bao, and H. Li. 2020. “Optimal policy for structure maintenance: A deep reinforcement learning framework.” Struct. Saf. 83 (Mar): 101906. https://doi.org/10.1016/j.strusafe.2019.101906.
Zhu, B., and D. M. Frangopol. 2016. “Time-variant risk assessment of bridges with partially and fully closed lanes due to traffic loading and scour.” J. Bridge Eng. 21 (6): 04016021. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000817.
Zulifqar, A., M. Cabieses, A. Mikhail, and N. Khan. 2014. Design of a bridge inspection system (BIS) to reduce time and cost. Fairfax, VA: George Mason Univ.
Information & Authors
Information
Published In
Journal of Computing in Civil Engineering
Volume 35 • Issue 6 • November 2021
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Mar 19, 2021
Accepted: Jul 9, 2021
Published online: Aug 30, 2021
Published in print: Nov 1, 2021
Discussion open until: Jan 30, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- Minghui Cheng, Dan M. Frangopol, Efficient scenario analysis for optimal adaptation of bridge networks under deep uncertainties through knowledge transfer, Structural Safety, 10.1016/j.strusafe.2022.102278, 100, (102278), (2023).
- Caio Filipe de Lima Munguba, Gustavo de Novaes Pires Leite, Alvaro Antonio Villa Ochoa, Enrique Lopez Droguett, Condition-based maintenance with reinforcement learning for refrigeration systems with selected monitored features, Engineering Applications of Artificial Intelligence, 10.1016/j.engappai.2023.106067, 122, (106067), (2023).
- Silvia Bianchi, Fabio Biondini, Life-Cycle Assessment of Deteriorating RC Bridges Using Artificial Neural Networks, Journal of Infrastructure Systems, 10.1061/(ASCE)IS.1943-555X.0000672, 28, 2, (2022).
- Jianda Cheng, Yan Liu, Minghui Cheng, Wei Li, Tianyun Li, Optimum condition-based maintenance policy with dynamic inspections based on reinforcement learning, Ocean Engineering, 10.1016/j.oceaneng.2022.112058, 261, (112058), (2022).