Incorporating Bayesian Networks in Markov Decision Processes
Publication: Journal of Infrastructure Systems
Volume 19, Issue 4
Abstract
This paper presents an extension to a partially observable Markov decision process so that its solution can take into account, at the beginning of the planning, the possible availability of free information in future time periods. It is assumed that such information has a Bayesian network structure. The proposed approach requires a smaller computational effort than the classical approaches used to solve dynamic Bayesian networks. Furthermore, it allows the user to (1) take advantage of prior probability distributions of relevant random variables that do not necessarily have a direct causal relationship with the state of the system; and (2) rationally take into account the effects of accidental or rare events (such as seismic activities) that may occur during future time periods of the planning horizon. The methodology is illustrated through an example problem that concerns the optimization of inspection, maintenance, and rehabilitation strategies of road pavement over a 14-year planning horizon.
Get full access to this article
View all available purchase options and get full access to this article.
References
AASHTO. (2011). Bridge element inspection manual, 1st Ed., Washington, DC, 172.
Attoh-Okine, N. O., and Bowers, S. (2006). “A Bayesian belief network model of bridge deterioration.” Proc. ICE—Bridge Eng., 159(2), 69–76.
Bellman, R. (1961). Adaptive control processes: A guided tour, Princeton University Press, Princeton, NJ.
Celeux, F., Corset, A., Lannoy, B., and Ricard, G. (2006). “Designing a Bayesian network for preventive maintenance from expert opinions in a rapid and reliable way.” Reliab. Eng. Syst. Saf., 91(7), 849–856.
Cooper, G. F. (1988). “A method for using belief networks as influence diagrams.” 4th Workshop on Uncertainty in Artificial Intelligence, Corvallis Press, Albany, OR, 55–63.
Cooper, G. F. (1990). “Probabilistic inference using Bayesian belief networks.” Artif. Intell., 42(2–3), 393–405.
Cooper, G. F., and Herskovits, E. (1992). “A Bayesian method for the induction of probabilistic networks from data.” Mach. Learn., 9(4), 309–347.
Corotis, R. B., Ellis, J. H., and Jiang, M. (2005). “Modeling of risk-based inspection, maintenance and life-cycle cost with partially observable Markov decision process.” Struct. Infrastruct. Eng., 1(1), 75–84.
Eckels, J. E. (1968). “Optimum maintenance with incomplete information.” Oper. Res., 16(5), 1058–1067.
Faddoul, R., Raphael, W., and Chateauneuf, A. (2011). “A generalized partially observable Markov decision process updated by decision trees for maintenance optimization.” Struct. Infrastruct. Eng., 7(10), 783–798.
Faddoul, R., Raphael, W., Chateauneuf, A., and Soubra, A. (2009a). “Optimal planning of inspection and maintenance by including epistemic uncertainties.” Proc., 1st Int. Conf. on Soft Computing Technology in Civil, Structural and Environmental Engineering, Civil-Comp Press, Stirlingshire, UK.
Faddoul, R., Raphael, W., Chateauneuf, A., and Soubra, A.-H. (2009b). “Processus généralisé de décision markovien partiellement observable utilisant les arbres de décisions.” Proc., 41st Journées de Statistique, Société Française de statistiques (SFdS), Paris (in French).
Faddoul, R., Raphael, W., Chateauneuf, A., and Soubra, A. (2010). “Processus de décisions markoviens parallèles soumis à des contraintes budgétaires.” JFMS’10, Hermes Science Publishing Ltd, London, UK (in French).
Faddoul, R., Soubra, A.-H., Raphael, W., and Chateauneuf, A. (2013). “Extension of dynamic programming models for management optimization from single structure to multi-structures level.” Struct. Infrastruct. Eng., 9(5), 432–447.
Federal Highway Administration. (1997). “Investigation of development of pavement roughness.”, Washington, DC.
Federal Highway Administration. (2008). “2008 Status of the nation’s highways, bridges, and transit: Conditions and performance.” Rep. to Congress, Washington, DC.
Francois, O., Bouillaut, L., Aknin, P., Leray, P., and Dubois, S. (2008). “Dynamic Bayesian network modeling maintenance strategies: Prevention of broken rails.” World Congress on Railway Research (WCRR’08), World Congress on Railway Research, Sydney.
Frangopol, D. M., Saydam, D., and Kim, S. (2012). “Maintenance, management, life-cycle design and performance of structures and infrastructures: A brief review.” Struct. Infrastruct. Eng., 8(1), 1–25.
Heckerman, D., Geiger, D., and Chickering, D. M. (1995). “Learning Bayesian networks: The combination of knowledge and statistical data.” Mach. Learn., 20(3), 197–243.
Hudson, W., Elkins, Cl., Uddin, W., and Reilly, K. (1987). “Improved methods and equipment to conduct pavement distress surveys.”, FHWA, Washington, DC.
Jensen, F., Jensen, F. V., and Dittmer, S. L. (1994). “From influence diagrams to junction trees.” Proc., 10th Conf. on Uncertainty in Artificial Intelligence, Morgan Kaufmann, San Francisco, CA, 367–373.
Jones, B., Jenkinson, I., Yang, Z., and Wang, J. (2010). “The use of Bayesian network modeling for maintenance planning in a manufacturing industry.” Reliab. Eng. Syst. Saf., 95(3), 267–277.
Langseth, H., and Portinale, L. (2007). “Bayesian networks in reliability.” Reliab. Eng. Syst. Saf., 92(1), 92–108.
Madanat, S. (1993). “Optimal infrastructure management decisions under uncertainty.” Transp. Res. Part C, 1(1), 71–80.
Madanat, S., and Ben-Akiva, M. (1994). “Optimal inspections and repair policies for infrastructure facilities.” Transp. Sci., 28(1), 55–62.
Manoj, K. J. (2009). “Dynamic Bayesian network for predicting the likelihood of a terrorist attack at critical transportation infrastructure facilities.” J. Infrastruct. Syst., 31–39.
Monahan, G. E. (1982). “A survey of partially observable Markov decision processes; Theory, models, and algorithms.” Manage. Sci., 28(1), 1–16.
Papadimitrious, C. H., and Tsitsiklis, J. N. (1987). “The complexity of Markov decision processes.” Math. Oper. Res., 12(3), 441–450.
Pearl, J. (1986). “Fusion, propagation and structuring in belief networks.” Artif. Intell., 29(3), 241–288.
Pearl, J., and Dechter, R. (1996). “Identifying independence in causal graphs with feedback.” Proc., 12th Annual Conf. on Uncertainty in Artificial Intelligence, Morgan and Kaufmann publishers, Burlington, MA, 420–426.
Robelin, C. A., and Madanat, S. (2007). “History-dependent optimization of bridge maintenance and replacement decisions using Markov decision processes.” J. Infrastruct. Syst., 195–201.
Sprites, P. (1995). “Directed cyclic graphical representations of feedback models.” Proc., 11th Annual Conf. on Uncertainty in Artificial Intelligence, Morgan and Kaufmann publishers, Burlington, MA, 491–498.
Sprites, P., Glymour, C., and Scheines, R. (1993). “Causation, prediction and search.” Lecture Notes in Statistics, Vol. 81, Springer-Verlag, Berlin.
Straub, D. (2009). “Stochastic modeling of deterioration processes through DBN.” J. Eng. Mech., 1089–1099.
Zhang, N. L. (1998). “Probabilistic inference in influence diagrams.” Comput. Intell., 14(4), 475–497.
Zhang, N. L., Qi, R., and Poole, D. (1992). “Stepwise-decomposable influence diagrams.” Proc., 3rd Conf. on Knowledge Representation, Cambridge, MA.
Zhang, N. L., Qi, R., and Poole, D. (1994). “A computational theory of decision networks.” Int. J. Approximate Reasoning, 11(2), 83–158.
Information & Authors
Information
Published In
Journal of Infrastructure Systems
Volume 19 • Issue 4 • December 2013
Pages: 415 - 424
Copyright
© 2013 American Society of Civil Engineers.
History
Received: Feb 1, 2012
Accepted: Oct 12, 2012
Published online: Oct 13, 2012
Discussion open until: Mar 13, 2013
Published in print: Dec 1, 2013
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.