Sustainable Life-Cycle Maintenance Policymaking for Network-Level Deteriorating Bridges with a Convolutional Autoencoder–Structured Reinforcement Learning Agent
Publication: Journal of Bridge Engineering
Volume 28, Issue 9
Abstract
Bridges play a significant role in urban areas, and their performance and safety are highly related to the carbon emissions of infrastructure systems. Previous studies have mainly offered maintenance policies that balance structural safety with overall costs. Considering the goal of achieving near-zero global carbon emissions by 2050, this study proposes a policymaking agent based on a convolutional autoencoder–structured deep-Q network (ConvAE-DQN) for managing deteriorating bridges at the network level while considering sustainability performance. This agent considers environmental, economic, and safety metrics, including spatially correlated structural failure probability, traffic volume, bridge size, and others, which are transformed into a multiattribute utility model to form the reward function. Reinforcement learning is employed to optimize the life-cycle maintenance planning to minimize the total carbon emissions and economic costs while maximizing regional safety performance. The proposed method is substantiated by developing sustainable life-cycle maintenance policies for an existing bridge network in Northern China. It is found that the proposed ConvAE-DQN policymaking agent could output efficient and sustainable life-cycle maintenance policies, which are annually stable and easy to schedule. The utility-based reward function enhances the stability and convergence efficiency of the algorithm. This study also assesses the impact of budget levels on network-level bridge safety and carbon footprint.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
The study has been supported by the Research Grant Council of Hong Kong (Project Nos. PolyU 15219819 and 15221521) and the Centrally Funded Postdoctoral Fellowship Scheme (P0043893). The support is gratefully acknowledged. The opinions and conclusions presented in this paper are those of the authors and do not necessarily reflect the views of the sponsoring organizations.
References
Abdallah, A. M., R. A. Atadero, and M. E. Ozbek. 2022. “A state-of-the-art review of bridge inspection planning: Current situation and future needs.” J. Bridge Eng. 27 (2): 03121001. https://doi.org/10.1061/(asce)be.1943-5592.0001812.
Abdelmaksoud, A. M., G. P. Balomenos, and T. C. Becker. 2021. “Parameterized logistic models for bridge inspection and maintenance scheduling.” J. Bridge Eng. 26 (10): 04021072. https://doi.org/10.1061/(asce)be.1943-5592.0001774.
Akiyama, M., D. M. Frangopol, and H. Ishibashi. 2020. “Toward life-cycle reliability-, risk- and resilience-based design and assessment of bridges and bridge networks under independent and interacting hazards: Emphasis on earthquake, tsunami and corrosion.” Struct. Infrastruct. Eng. 16 (1): 26–50. https://doi.org/10.1080/15732479.2019.1604770.
Akiyama, M., D. M. Frangopol, and H. Matsuzaki. 2011. “Life-cycle reliability of RC bridge piers under seismic and airborne chloride hazards.” Earthquake Eng. Struct. Dyn. 40 (15): 1671–1687. https://doi.org/10.1002/eqe.1108.
Alampalli, S., D. M. Frangopol, J. Grimson, M. W. Halling, D. E. Kosnik, E. O. L. Lantsoght, D. Yang, and Y. E. Zhou. 2021. “Bridge load testing: State-of-the-practice.” J. Bridge Eng. 26 (3): 03120002. https://doi.org/10.1061/(asce)be.1943-5592.0001678.
Alipour, M., D. K. Harris, L. E. Barnes, O. E. Ozbulut, and J. Carroll. 2017. “Load-capacity rating of bridge populations through machine learning: Application of decision trees and random forests.” J. Bridge Eng. 22 (10): 04017076. https://doi.org/10.1061/(ASCE)BE.1943-5592.0001103.
Andriotis, C. P., and K. G. Papakonstantinou. 2019. “Managing engineering systems with large state and action spaces through deep reinforcement learning.” Reliab. Eng. Syst. Saf. 191: 106483. https://doi.org/10.1016/j.ress.2019.04.036.
Andriotis, C. P., and K. G. Papakonstantinou. 2021. “Deep reinforcement learning driven inspection and maintenance planning under incomplete information and constraints.” Reliab. Eng. Syst. Saf. 212: 107551. https://doi.org/10.1016/j.ress.2021.107551.
Bianchi, S., and F. Biondini. 2022. “Life-cycle assessment of deteriorating RC bridges using artificial neural networks.” J. Infrastruct. Syst. 28 (2): 04022005. https://doi.org/10.1061/(asce)is.1943-555x.0000672.
Bocchini, P., D. M. Frangopol, T. Ummenhofer, and T. Zinke. 2014. “Resilience and sustainability of civil infrastructure: Toward a unified approach.” J. Infrastruct. Syst. 20 (2): 04014004. https://doi.org/10.1061/(asce)is.1943-555x.0000177.
Busoniu, L., R. Babuska, and B. De Schutter. 2008. “A comprehensive survey of multiagent reinforcement learning.” IEEE Trans. Syst. Man Cybern. Part C Appl. Rev. 38 (2): 156–172. https://doi.org/10.1109/tsmcc.2007.913919.
Chen, S. Z., and D. C. Feng. 2022. “Multifidelity approach for data-driven prediction models of structural behaviors with limited data.” Comput.-Aided Civ. Infrastruct. Eng. 37 (12): 1566–1581. https://doi.org/10.1111/mice.12817.
Cheng, M. H., and D. M. Frangopol. 2021. “A decision-making framework for load rating planning of aging bridges using deep reinforcement learning.” J. Comput. Civil Eng. 35 (6): 04021024. https://doi.org/10.1061/(asce)cp.1943-5487.0000991.
Cheng, M. H., and D. M. Frangopol. 2022. “Life-cycle optimization of structural systems based on cumulative prospect theory: Effects of the reference point and risk attitudes.” Reliab. Eng. Syst. Saf. 218: 108100. https://doi.org/10.1016/j.ress.2021.108100.
Cheng, M. H., D. M. Frangopol, and C. Q. Gong. 2020a. “Acceptance of corrosion-resistant steel in design of steel girder bridges based on expected utility theory.” J. Bridge Eng. 25 (11): 04020098. https://doi.org/10.1061/(asce)be.1943-5592.0001627.
Cheng, M. H., D. Y. Yang, and D. M. Frangopol. 2020b. “Investigation of effects of time preference and risk perception on life-cycle management of civil infrastructure.” ASCE-ASME J. Risk Uncertainty Eng. Syst. Part A: Civ. Eng. 6 (1): 04020001. https://doi.org/10.1061/ajrua6.0001039.
Dong, Y., and D. M. Frangopol. 2017. “Probabilistic assessment of an interdependent healthcare-bridge network system under seismic hazard.” Struct. Infrastruct. Eng. 13 (1): 160–170. https://doi.org/10.1080/15732479.2016.1198399.
Dong, Y., D. M. Frangopol, and D. Saydam. 2014. “Sustainability of highway bridge networks under seismic hazard.” J. Earthquake Eng. 18 (1): 41–66. https://doi.org/10.1080/13632469.2013.841600.
Du, A., and A. Ghavidel. 2022. “Parameterized deep reinforcement learning-enabled maintenance decision-support and life-cycle risk assessment for highway bridge portfolios.” Struct. Saf. 97: 102221. https://doi.org/10.1016/j.strusafe.2022.102221.
Du, G. L., M. Safi, L. Pettersson, and R. Karoumi. 2014. “Life cycle assessment as a decision support tool for bridge procurement: Environmental impact comparison among five bridge designs.” Int. J. Life Cycle Assess. 19 (12): 1948–1964. https://doi.org/10.1007/s11367-014-0797-z.
Fiorillo, G., and M. Ghosn. 2022. “Risk-based life-cycle analysis of highway bridge networks under budget constraints.” Struct. Infrastruct. Eng. 18(10–11): 1457–1471. https://doi.org/10.1080/15732479.2022.2059525.
Frangopol, D. M., and M. Liu. 2007. “Maintenance and management of civil infrastructure based on condition, safety, optimization, and life-cycle cost.” Struct. Infrastruct. Eng. 3 (1): 29–41. https://doi.org/10.1080/15732470500253164.
Galvão, N., J. C. Matos, D. V. Oliveira, and R. Hajdin. 2021. “Human error impact in structural safety of a reinforced concrete bridge.” Struct. Infrastruct. Eng. 18 (6): 836–850. https://doi.org/10.1080/15732479.2021.1876105.
Gokasar, I., M. Deveci, and O. Kalan. 2022. “CO2 emission based prioritization of bridge maintenance projects using neutrosophic fuzzy sets based decision making approach.” Res. Transp. Econ. 91: 101029. https://doi.org/10.1016/j.retrec.2021.101029.
Han, X., and D. M. Frangopol. 2023. “Life-cycle risk-based optimal maintenance strategy for bridge networks subjected to corrosion and seismic hazards.” J. Bridge Eng. 28 (1): 04022128. https://doi.org/10.1061/JBENF2.BEENG-5799.
Iacovino, C., Z. I. Turksezer, P. F. Giordano, and M. P. Limongelli. 2022. “Comparison of bridge inspection policies in terms of data quality.” J. Bridge Eng. 27 (3): 04021115. https://doi.org/10.1061/(asce)be.1943-5592.0001831.
Khan, M. S., C. Caprani, S. Ghosh, and J. Ghosh. 2022. “Value of strain-based structural health monitoring as decision support for heavy load access to bridges.” Struct. Infrastruct. Eng. 18 (4): 521–536. https://doi.org/10.1080/15732479.2021.1890140.
Lei, X., Y. Xia, A. Wang, X. Jian, H. Zhong, and L. Sun. 2023. “Mutual information based anomaly detection of monitoring data with attention mechanism and residual learning.” Mech. Syst. Sig. Process. 182: 109607. https://doi.org/https://doi.org/10.1016/j.ymssp.2022.109607.
Lei, X. M., and Y. Dong. 2022. “Deep reinforcement learning for optimal life-cycle management of deteriorating regional bridges using double-deep Q-networks.” Smart Struct. Syst. 30 (6): 571–582. https://doi.org/10.12989/sss.2022.30.6.571.
Lei, X. M., L. M. Sun, and Y. Xia. 2021a. “Lost data reconstruction for structural health monitoring using deep convolutional generative adversarial networks.” Struct. Health Monit. 20 (4): 2069–2087. https://doi.org/10.1177/1475921720959226.
Lei, X. M., L. M. Sun, and Y. Xia. 2021b. “Seismic fragility assessment and maintenance management on regional bridges using Bayesian multi-parameter estimation.” Bull. Earthquake Eng. 19: 6693–6717. https://doi.org/10.1007/s10518-021-01072-6.
Lei, X. M., Y. Xia, L. Deng, and L. M. Sun. 2022. “A deep reinforcement learning framework for life-cycle maintenance planning of regional deteriorating bridges using inspection data.” Struct. Multidiscip. Optim. 65 (5): 149. https://doi.org/10.1007/s00158-022-03210-3.
Liu, H., and Y. F. Zhang. 2020. “Bridge condition rating data modeling using deep learning algorithm.” Struct. Infrastruct. Eng. 16 (10): 1447–1460. https://doi.org/10.1080/15732479.2020.1712610.
Liu, Y. S., Y. F. Wang, C. C. Shi, W. J. Zhang, W. Luo, J. J. Wang, K. P. Li, N. Yeung, and S. Kite. 2022. “Assessing the CO2 reduction target gap and sustainability for bridges in China by 2040.” Renewable Sustainable Energy Rev. 154: 111811. https://doi.org/10.1016/j.rser.2021.111811.
Memarzadeh, M., and M. Pozzi. 2019. “Model-free reinforcement learning with model-based safe exploration: Optimizing adaptive recovery process of infrastructure systems.” Struct. Saf. 80: 46–55. https://doi.org/10.1016/j.strusafe.2019.04.003.
Miyamoto, A., and M. Motoshita. 2015. “Development and practical application of a bridge management system (J-BMS) in Japan.” Civ. Eng. Infrastruct. J. 20 (4): 209–221. https://doi.org/10.7508/CEIJ.2015.01.013.
Mnih, V., et al. 2015. “Human-level control through deep reinforcement learning.” Nature 518 (7540): 529–533. https://doi.org/10.1038/nature14236.
Morcous, G., and Z. Lounis. 2005. “Maintenance optimization of infrastructure networks using genetic algorithms.” Autom. Constr. 14 (1): 129–142. https://doi.org/10.1016/j.autcon.2004.08.014.
Mortagi, M., and J. Ghosh. 2022. “Consideration of climate change effects on the seismic life-cycle cost analysis of deteriorating highway bridges.” J. Bridge Eng. 27 (2): 04021103. https://doi.org/10.1061/(asce)be.1943-5592.0001815.
Omer, M., L. Margetts, M. H. Mosleh, and L. S. Cunningham. 2021. “Inspection of concrete bridge structures: Case study comparing conventional techniques with a virtual reality approach.” J. Bridge Eng. 26 (10): 05021010. https://doi.org/10.1061/(asce)be.1943-5592.0001759.
Peng, J. X., Y. M. Yang, H. B. Bian, J. R. Zhang, and L. Wang. 2022. “Optimisation of maintenance strategy of deteriorating bridges considering sustainability criteria.” Struct. Infrastruct. Eng. 18 (3): 395–411. https://doi.org/10.1080/15732479.2020.1855215.
Sabatino, S., D. M. Frangopol, and Y. Dong. 2015. “Sustainability-informed maintenance optimization of highway bridges considering multi-attribute utility and risk attitude.” Eng. Struct. 102: 310–321. https://doi.org/10.1016/j.engstruct.2015.07.030.
Scope, C., M. Vogel, and E. Guenther. 2021. “Greener, cheaper, or more sustainable: Reviewing sustainability assessments of maintenance strategies of concrete structures.” Sustainable Prod. Consumption 26: 838–858. https://doi.org/10.1016/j.spc.2020.12.022.
Thompson, P. D., E. P. Small, M. Johnson, and A. R. Marshall. 1998. “The Pontis bridge management system.” Struct. Eng. Int. 8 (4): 303–308. https://doi.org/10.2749/101686698780488758.
Val, D. V., and M. G. Stewart. 2003. “Life-cycle cost analysis of reinforced concrete structures in marine environments.” Struct. Saf. 25 (4): 343–362. https://doi.org/10.1016/s0167-4730(03)00014-6.
Vishnu, N., S. Kameshwar, and J. E. Padgett. 2021. “Road transportation network hazard sustainability and resilience: Correlations and comparisons.” Struct. Infrastruct. Eng. 19 (3): 345–365. https://doi.org/10.1080/15732479.2021.1945114.
Wang, F. Y., C. C. Lee, and N. G. Gharaibeh. 2022. “Network-level bridge deterioration prediction models that consider the effect of maintenance and rehabilitation.” J. Infrastruct. Syst. 28 (1): 05021009. https://doi.org/10.1061/(asce)is.1943-555x.0000662.
Wanniarachchi, S., T. Prabatha, H. Karunathilake, S. Li, M. S. Alam, and K. Hewage. 2022a. “Life cycle thinking-based decision making for bridges under seismic conditions. II: A case study on bridges with superelastic SMA RC piers.” J. Bridge Eng. 27 (6): 04022043. https://doi.org/10.1061/(asce)be.1943-5592.0001885.
Wanniarachchi, S., T. Prabatha, H. Karunathilake, Q. Zhang, K. Hewage, and M. S. Alam. 2022b. “Life cycle thinking-based decision making for bridges under seismic conditions. I: Methodology and framework.” J. Bridge Eng. 27 (6): 04022044. https://doi.org/10.1061/(asce)be.1943-5592.0001884.
Xia, Y., X. M. Lei, P. Wang, and L. M. Sun. 2021. “Artificial intelligence based structural assessment for regional short- and medium-span concrete beam bridges with inspection information.” Remote Sens. 13 (18): 3687. https://doi.org/10.3390/rs13183687.
Xia, Y., X. M. Lei, P. Wang, and L. M. Sun. 2022. “A data-driven approach for regional bridge condition assessment using inspection reports.” Struct. Control Health Monit. 29 (4): e2915. https://doi.org/10.1002/stc.2915.
Xin, J. Y., M. Akiyama, D. M. Frangopol, and M. Y. Zhang. 2022. “Multi-objective optimisation of in-service asphalt pavement maintenance schedule considering system reliability estimated via LSTM neural networks.” Struct. Infrastruct. Eng. 18 (7): 1002–1019. https://doi.org/10.1080/15732479.2022.2038641.
Xu, G. W., and F. D. Guo. 2022. “Sustainability-oriented maintenance management of highway bridge networks based on Q-learning.” Sustainable Cities Soc. 81: 103855. https://doi.org/10.1016/j.scs.2022.103855.
Xu, J. G., Z. K. Cai, and D. C. Feng. 2021. “Life-cycle seismic performance assessment of aging RC bridges considering multi-failure modes of bridge columns.” Eng. Struct. 244: 112818. https://doi.org/10.1016/j.engstruct.2021.112818.
Xu, J. G., D. C. Feng, S. Mangalathu, and J. S. Jeon. 2022. “Data-driven rapid damage evaluation for life-cycle seismic assessment of regional reinforced concrete bridges.” Earthquake Eng. Struct. Dyn. 51 (11): 2730–2751. https://doi.org/10.1002/eqe.3699.
Yang, D. Y., and D. M. Frangopol. 2018. “Risk-informed bridge ranking at project and network levels.” J. Infrastruct. Syst. 24 (3): 04018018. https://doi.org/10.1061/(asce)is.1943-555x.0000430.
Yao, X. F., M. G. Zhai, Y. Xu, J. Z. Zhang, and Y. J. Ding. 2008. “Study on safety degree for existing concrete girder bridge members based on calculated reliability.” In Proc., 14th World Conf. on Earthquake Engineering. Tokyo, Japan: International Association for Earthquake Engineering (IAEE).
Yosri, A., Y. Elleathy, S. Hassini, and W. El-Dakhakhni. 2021. “Genetic algorithm-Markovian model for predictive bridge asset management.” J. Bridge Eng. 26 (8): 04021052. https://doi.org/10.1061/(asce)be.1943-5592.0001752.
Zhang, N. L., and W. J. Si. 2020. “Deep reinforcement learning for condition-based maintenance planning of multi-component systems under dependent competing risks.” Reliab. Eng. Syst. Saf. 203: 107094. https://doi.org/10.1016/j.ress.2020.107094.
Zhao, F. Q., F. Q. Liu, Z. W. Liu, and H. Hao. 2019. “The correlated impacts of fuel consumption improvements and vehicle electrification on vehicle greenhouse gas emissions in China.” J. Cleaner Prod. 207: 702–716. https://doi.org/10.1016/j.jclepro.2018.10.046.
Zhou, Z. Y., L. Lai, and Y. Dong. 2022. “Quantification of value of information associated with optimal observation actions within partially observable Markov decision processes.” KSCE J. Civ. Eng. 26 (12): 5173–5186. https://doi.org/10.1007/s12205-022-2121-y.
Information & Authors
Information
Published In
Journal of Bridge Engineering
Volume 28 • Issue 9 • September 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Nov 5, 2022
Accepted: May 4, 2023
Published online: Jul 5, 2023
Published in print: Sep 1, 2023
Discussion open until: Dec 5, 2023
ASCE Technical Topics:
- Aging (material)
- Air pollution
- Architectural engineering
- Bridge engineering
- Bridge management
- Bridge tests
- Building management
- Business management
- Design (by type)
- Deterioration
- Emissions
- Engineering fundamentals
- Environmental engineering
- Field tests
- Life cycles
- Maintenance and operation
- Materials characterization
- Materials engineering
- Pollution
- Practice and Profession
- Structural design
- Structural engineering
- Structural safety
- Sustainable development
- Tests (by type)
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.