Adaptive Risk-Based Life-Cycle Management for Large-Scale Structures Using Deep Reinforcement Learning and Surrogate Modeling
Publication: Journal of Engineering Mechanics
Volume 148, Issue 1
Abstract
Optimal life-cycle management is a challenging task for large-scale structures. The complexity of structural states, represented by the numerous combinations of component conditions, and the vast number of inspection and maintenance options often prompt the decision-makers to adopt a simple time- or condition-based management method rather than a performance-based one. To improve this situation, this study proposes a novel method for adaptive risk-based life-cycle management of large-scale structures. The proposed method can yield bespoke inspection and maintenance plans at the individual component level based on their contribution to the overall structural performance. The obtained plan can also adapt itself to the unfolding information gained from inspection and maintenance actions. This advanced method, termed DeepLCM, is enabled by (1) efficient surrogate modeling based on deep neural networks for structural risk assessment; and (2) a deep reinforcement learning algorithm for adaptive life-cycle management. The method is applied to a steel girder bridge in Montgomery County, Pennsylvania. The inspection and maintenance plan obtained using DeepLCM is compared with those obtained using the conventional life-cycle management techniques including time-, condition-, and risk-based methods. The case study also investigates the effect of the spatial granularity of inspection and maintenance actions on the resulting life-cycle cost.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
Some or all data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request. Specifically, the training data for the deep neural network and the trained actor and critic networks are available upon request.
Acknowledgments
The author is grateful for the financial support received from Portland State University. The opinions and conclusions presented in this paper are those of the author and do not necessarily reflect the views of the sponsoring organization.
References
AASHTO. 2018. The manual for bridge evaluation. Washington, DC: AASHTO.
Abadi, M., et al. 2015. “TensorFlow: Large-scale machine learning on heterogeneous systems.” Accessed December 31, 2019. https://www.tensorflow.org/.
Albrecht, P., and A. H. Naeemi. 1984. NCHRP report 272: Performance of weathering steel in bridges. Washington, DC: Transportation Research Board.
Andriotis, C. P., and K. G. Papakonstantinou. 2019. “Managing engineering systems with large state and action spaces through deep reinforcement learning.” Reliab. Eng. Syst. Saf. 191 (Nov): 106483. https://doi.org/10.1016/j.ress.2019.04.036.
Blatman, G., and B. Sudret. 2010. “An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis.” Probab. Eng. Mech. 25 (2): 183–197. https://doi.org/10.1016/j.probengmech.2009.10.003.
Chojaczyk, A. A., A. P. Teixeira, L. C. Neves, J. B. Cardoso, and C. Guedes Soares. 2015. “Review and application of artificial neural networks models in reliability analysis of steel structures.” Struct. Saf. 52 (Jan): 78–89. https://doi.org/10.1016/j.strusafe.2014.09.002.
Chollet, F. 2015. “Keras.” Accessed February 8, 2021. https://keras.io.
Corotis, R. B., J. Hugh Ellis, and M. Jiang. 2005. “Modeling of risk-based inspection, maintenance and life-cycle cost with partially observable Markov decision processes.” Struct. Infrastruct. Eng. 1 (1): 75–84. https://doi.org/10.1080/15732470412331289305.
Czarnecki, A. A., and A. S. Nowak. 2008. “Time-variant reliability profiles for steel girder bridges.” Struct. Saf. 30 (1): 49–64. https://doi.org/10.1016/j.strusafe.2006.05.002.
Dewey, D. 2014. “Reinforcement learning and the reward engineering principle.” In Proc., Implementing Selves with Safe Motivational Systems and Motivational Systems and Self-Improvement: Papers from the AAAI Spring Symp., 13–16. Menlo Park, CA: Association for the Advancement of Artificial Intelligence.
DNV GL (Det Norske Veritas and Germanischer Lloyd). 2014. Inspection and maintenance of jacking systems. Høvik, Norway: DNV GL.
Dubourg, V., B. Sudret, and J.-M. Bourinet. 2011. “Reliability-based design optimization using kriging surrogates and subset simulation.” Struct. Multidiscip. Optim. 44 (5): 673–690. https://doi.org/10.1007/s00158-011-0653-8.
Ellingwood, B. R., T. V. Galambos, J. G. MacGregor, and C. A. Cornell. 1980. Development of a probability based load criterion for American National Standard A58: Building code requirements for minimum design loads in buildings and other structures. Washington, DC: National Bureau of Standards.
FHWA (Federal Highway Administration). 2012. Bridge inspector’s reference manual. Washington, DC: FHWA.
Fiorillo, G., and M. Ghosn. 2019. “Risk-based importance factors for bridge networks under highway traffic loads.” Struct. Infrastruct. Eng. 15 (1): 113–126. https://doi.org/10.1080/15732479.2018.1496119.
Frangopol, D. M., J. S. Kong, and E. S. Gharaibeh. 2001. “Reliability-based life-cycle management of highway bridges.” J. Comput. Civ. Eng. 15 (1): 27–34. https://doi.org/10.1061/(ASCE)0887-3801(2001)15:1(27).
Frangopol, D. M., K.-Y. Lin, and A. C. Estes. 1997. “Life-cycle cost design of deteriorating structures.” J. Struct. Eng. 123 (10): 1390–1401. https://doi.org/10.1061/(ASCE)0733-9445(1997)123:10(1390).
Frangopol, D. M., and M. Soliman. 2016. “Life-cycle of structural systems: Recent achievements and future directions.” Struct. Infrastruct. Eng. 12 (1): 1–20. https://doi.org/10.1080/15732479.2014.999794.
Furuta, H., T. Kameda, K. Nakahara, Y. Takahashi, and D. M. Frangopol. 2006. “Optimal bridge maintenance planning using improved multi-objective genetic algorithm.” Struct. Infrastruct. Eng. 2 (1): 33–41. https://doi.org/10.1080/15732470500031040.
García-Segura, T., V. Yepes, D. M. Frangopol, and D. Y. Yang. 2017. “Lifetime reliability-based optimization of post-tensioned box-girder bridges.” Eng. Struct. 145 (Aug): 381–391. https://doi.org/10.1016/j.engstruct.2017.05.013.
Gehl, P., and D. D’Ayala. 2018. “System loss assessment of bridge networks accounting for multi-hazard interactions.” Struct. Infrastruct. Eng. 14 (10): 1355–1371.
Gelman, A., J. B. Carlin, H. S. Stern, D. B. Dunson, A. Vehtari, D. B. Rubin, D. B. Bunson, A. Vehtari, and D. B. Rubin. 2013. Bayesian data analysis. 3rd ed. New York: CRC Press.
Gervásio, H., L. S. da Silva, V. Perdigão, A. Orcesi, and R. Andersen. 2015. “Influence of maintenance strategies on the life cycle performance of composite highway bridges.” Struct. Eng. Int. 25 (2): 184–196. https://doi.org/10.2749/101686614X14043795569978.
Ghosn, M., G. Fiorillo, V. Gayovyy, T. Getso, S. Ahmed, and N. Parker. 2015. Effects of overweight vehicles on NYSDOT infrastructure. Technical Rep. NYSDOT C-08-13. Albany, NY: New York State DOT.
Ghosn, M., and F. Moses. 1998. Redundancy of bridge superstructure. NCHRP Rep. 406. Washington, DC: National Academy Press.
Haladuick, S., and M. R. Dann. 2018. “Value of information-based decision analysis of the optimal next inspection type for deteriorating structural systems.” Struct. Infrastruct. Eng. 14 (9): 1283–1292. https://doi.org/10.1080/15732479.2018.1433692.
Han, X., D. Y. Yang, and D. M. Frangopol. 2020. “Risk-based life-cycle optimization of deteriorating steel bridges: Investigation on the use of novel corrosion resistant steel.” Adv. Struct. Eng. 24 (8): 1668–1686. https://doi.org/10.1177/1369433220980529.
IACS (International Association of Classification Societies). 2012. Common structural rules for double hull oil tankers. London: IACS.
Kim, T., O.-S. Kwon, and J. Song. 2019. “Response prediction of nonlinear hysteretic systems by deep neural networks.” Neural Netw. 111 (2): 1–10. https://doi.org/10.1016/j.neunet.2018.12.005.
Kumar, R., P. Gardoni, and M. Sánchez-Silva. 2009. “Effect of cumulative seismic damage and corrosion on the life-cycle cost of reinforced concrete bridges.” Earthquake Eng. Struct. Dyn. 38 (7): 887–905. https://doi.org/10.1002/eqe.873.
Liu, L., D. Y. Yang, and D. M. Frangopol. 2020. “Network-level risk-based framework for optimal bridge adaptation management considering scour and climate change.” J. Infrastruct. Syst. 26 (1): 04019037. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000516.
Melchers, R. E. 1999. Structural reliability: Analysis and prediction. Chichester, UK: Wiley.
Memarzadeh, M., and M. Pozzi. 2016. “Value of information in sequential decision making: Component inspection, permanent monitoring and system-level scheduling.” Reliab. Eng. Syst. Saf. 154 (5): 137–151. https://doi.org/10.1016/j.ress.2016.05.014.
Memarzadeh, M., and M. Pozzi. 2019. “Model-free reinforcement learning with model-based safe exploration: Optimizing adaptive recovery process of infrastructure systems.” Struct. Saf. 80 (Sep): 46–55. https://doi.org/10.1016/j.strusafe.2019.04.003.
Memarzadeh, M., M. Pozzi, and J. Zico Kolter. 2015. “Optimal planning and learning in uncertain environments for the management of wind farms.” J. Comput. Civ. Eng. 29 (5): 04014076. https://doi.org/10.1061/(ASCE)CP.1943-5487.0000390.
Mnih, V., et al. 2015. “Human-level control through deep reinforcement learning.” Nature 518 (7540): 529–533. https://doi.org/10.1038/nature14236.
Mnih, V., A. P. Badia, M. Mirza, A. Graves, T. P. Lillicrap, T. Harley, D. Silver, and K. Kavukcuoglu. 2016. “Asynchronous methods for deep reinforcement learning.” Preprint, submitted June 16, 2016. http://arxiv.org/abs/1602.01783.
Mondoro, A., M. Soliman, and D. M. Frangopol. 2016. “Prediction of structural response of naval vessels based on available structural health monitoring data.” Ocean Eng. 125 (Apr): 295–307. https://doi.org/10.1016/j.oceaneng.2016.08.012.
Nielson, B. G. 2005. Analytical fragility curves for highway bridges in moderate seismic zones. Atlanta: Georgia Institute of Technology.
OpenSees. 2019. Open system for earthquake engineering simulation (OpenSees) Version 3.1.0 [Computer Software]. Berkeley, CA: Pacific Earthquake Engineering Research Center.
Papadrakakis, M., and N. D. Lagaros. 2002. “Reliability-based structural optimization using neural networks and Monte Carlo simulation.” Comput. Methods Appl. Mech. Eng. 191 (32): 3491–3507. https://doi.org/10.1016/S0045-7825(02)00287-6.
Papakonstantinou, K. G., and M. Shinozuka. 2014. “Optimum inspection and maintenance policies for corroded structures using partially observable Markov decision processes and stochastic, physically based models.” Probab. Eng. Mech. 37 (Jul): 93–108. https://doi.org/10.1016/j.probengmech.2014.06.002.
Silver, D., G. Lever, N. Heess, T. Degris, D. Wierstra, and M. Riedmiller. 2014. “Deterministic policy gradient algorithms.” In Proc., 31st Int. Conf. on Machine Learning, 605–619. Princeton, NJ: International Machine Learning Society.
Sobanjo, J. O., and P. D. Thompson. 2001. Development of agency maintenance, repair & rehabilitation (MR&R) cost data for Florida’s bridge management system. Tallahassee, FL: Florida DOT.
Soliman, M., and D. M. Frangopol. 2014. “Life-cycle management of fatigue-sensitive structures integrating inspection information.” J. Infrastruct. Syst. 20 (2): 04014001. https://doi.org/10.1061/(ASCE)IS.1943-555X.0000169.
Straub, D., J. D. Sørensen, J. Goyet, and M. H. Faber. 2006. “Benefits of risk based inspection planning for offshore structures.” In Proc., Int. Conf. on Offshore Mechanics and Arctic Engineering (OMAE2006), 1–10. New York: American Society of Mechanical Engineers.
Sundar, V. S., and M. D. Shields. 2016. “Surrogate-enhanced stochastic search algorithms to identify implicitly defined functions for reliability analysis.” Struct. Saf. 62 (Sep): 1–11. https://doi.org/10.1016/j.strusafe.2016.05.001.
Sutton, R. S., and A. G. Barto. 2018. Reinforcement learning: An introduction. 2nd ed. Cambridge, MA: MIT Press.
Sutton, R. S., D. McAllester, S. Singh, and Y. Mansour. 2000. “Policy gradient methods for reinforcement learning with function approximation.” In Advances in neural information processing systems, 1057–1063. Cambridge, MA: MIT Press.
Tao, Z., R. B. Corotis, and J. H. Ellis. 1994. “Reliability-based bridge design and life cycle management with Markov decision processes.” Struct. Saf. 16 (1–2): 111–132. https://doi.org/10.1016/0167-4730(94)00030-T.
Taucer, F. F., E. Spacone, and F. C. Filippou. 1991. A fiber beam-column element for seismic response analysis of reinforced concrete structures. Berkeley, CA: Univ. of California.
Van Hasselt, H., A. Guez, and D. Silver. 2015. “Deep reinforcement learning with double Q-learning.” Preprint, submitted December 8, 2015. https://arxiv.org/abs/1509.06461.
Van-Noortwijk, J. M., and D. M. Frangopol. 2004. “Two probabilistic life-cycle maintenance models for deteriorating civil infrastructures.” Probab. Eng. Mech. 19 (4): 345–359. https://doi.org/10.1016/j.probengmech.2004.03.002.
Wang, J. X., Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. 2016. “Learning to reinforcement learn.” Preprint, submitted January 23, 2017. https://arxiv.org/abs/1611.05763.
Wang, N., B. R. Ellingwood, and A.-H. Zureick. 2011. “Bridge rating using system reliability assessment. II: Improvements to bridge rating practices.” J. Bridge Eng. 16 (6): 863–871. https://doi.org/10.1061/(ASCE)BE.1943-5592.0000171.
Wang, Z., L. Dueñas-Osorio, and J. E. Padgett. 2013. “Seismic response of a bridge-soil-foundation system under the combined effect of vertical and horizontal ground motions.” Earthquake Eng. Struct. Dyn. 42 (4): 545–564.
Wang, Z., T. Schaul, M. Hessel, H. van Hasselt, M. Lanctot, and N. de Freitas. 2015. “Dueling network architectures for deep reinforcement learning.” Preprint, submitted April 5, 2016. https://arxiv.org/abs/1511.06581.
Wei, S., Y. Bao, and H. Li. 2020. “Optimal policy for structure maintenance: A deep reinforcement learning framework.” Struct. Saf. 83 (4): 101906. https://doi.org/10.1016/j.strusafe.2019.101906.
Williams, R. J. 1992. “Simple statistical gradient-following algorithms for connectionist reinforcement learning.” Mach. Learn. 8 (3–4): 229–256. https://doi.org/10.1007/BF00992696.
Wu, Y., E. Mansimov, S. Liao, R. Grosse, and J. Ba. 2017. “Scalable trust-region method for deep reinforcement learning using Kronecker-factored approximation.” Preprint, submitted August 17, 2017. https://arxiv.org/abs/1708.05144.
Yang, D. Y., and D. M. Frangopol. 2018a. “Evidence-based framework for real-time life-cycle management of fatigue-critical details of structures.” Struct. Infrastruct. Eng. 14 (5): 509–522. https://doi.org/10.1080/15732479.2017.1399150.
Yang, D. Y., and D. M. Frangopol. 2018b. “Probabilistic optimization framework for inspection/repair planning of fatigue-critical details using dynamic Bayesian networks.” Comput. Struct. 198 (5): 40–50. https://doi.org/10.1016/j.compstruc.2018.01.006.
Yang, D. Y., and D. M. Frangopol. 2021. “Risk-based inspection planning of deteriorating structures.” Struct. Infrastruct. Eng.
Yang, D. Y., D. M. Frangopol, and J.-G. Teng. 2019. “Probabilistic life-cycle optimization of durability-enhancing maintenance actions: Application to FRP strengthening planning.” Eng. Struct. 188 (Apr): 340–349. https://doi.org/10.1016/j.engstruct.2019.02.055.
Zhu, M., F. McKenna, and M. H. Scott. 2018. “OpenSeesPy: Python library for the OpenSees finite element framework.” SoftwareX 7 (Jan): 6–11. https://doi.org/10.1016/j.softx.2017.10.009.
Information & Authors
Information
Published In
Journal of Engineering Mechanics
Volume 148 • Issue 1 • January 2022
Copyright
© 2021 American Society of Civil Engineers.
History
Received: Feb 14, 2021
Accepted: Aug 23, 2021
Published online: Oct 27, 2021
Published in print: Jan 1, 2022
Discussion open until: Mar 27, 2022
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.
Cited by
- David Y. Yang, Deep Reinforcement Learning–Enabled Bridge Management Considering Asset and Network Risks, Journal of Infrastructure Systems, 10.1061/(ASCE)IS.1943-555X.0000704, 28, 3, (2022).