Multiagent Reinforcement Learning for Project-Level Intervention Planning under Multiple Uncertainties

Reinforcement learning (RL) has recently been adopted by infrastructure asset management (IAM) researchers for adding flexibility regarding uncertainties in preventive actions decision-making. However, this relatively recent line of research has not incorporated other sources of uncertainties, such as hazards apart from deterioration patterns, nor has such research considered managerial aspects of IAM, such as stakeholders’ utilities. This paper aims to provide a holistic framework that draws upon recent developments in IAM systems and microworlds, employs RL model training, and considers deterioration, hazards, and cost fluctuations as the main sources of uncertainties while also adopting managerial aspects into decision-making. Consistent with the existing practice of IAM, this framework brings flexibility in the face of uncertainties to the IAM decision-making process. Multi-agent RL models based on deep Q networks and actor-critic models are constructed and trained for taking intervention actions regarding elements of a real bridge in Indiana through its life cycle. Both models could lead to higher expected utilities and lower costs compared to the optimal maintenance, rehabilitation, and reconstruction (MRR) plans obtained by Monte Carlo simulation and heuristic optimization algorithms. The proposed framework can assist decision-making bodies and managers in the IAM domain with making updateable optimal and more realistic decisions based on the updated state of various complex uncertainties in a negligible amount of time.

Long-term strategic intervention actions are vital for maintaining the safety of degraded infrastructure assets. Traditionally, these strategic plans are derived through using optimization methods for maximizing stakeholders’ utilities, improving the safety of the network, and minimizing costs and risks associated with catastrophic events. The main drawback, and perhaps the biggest reported challenge, of this approach is the lack of flexibility in the face of the updated state of uncertain phenomena, such as the condition of assets and volatile costs. Simply put, the rare outcomes of unprecedented events can render optimal strategies useless. This study promotes the application of artificial intelligence-based agents for taking updateable decisions given the newly-observed state of all uncertain factors. Based on the results of the case study, the proposed framework was found able to reduce costs by up to 7% during the management horizon. This reduction in costs can be translated into saving millions of dollars of taxpayers’ money on a community scale. The theoretically well-founded and practically applicable proposed framework could measurably enhance the management of various infrastructure assets. Given that it can be tailored to other decision-making processes under uncertainty, the proposed framework can be similarly applied to other complex managerial problems.