Model Selection in Applied Science and Engineering: A Decision-Theoretic Approach
This article has a reply.
VIEW THE REPLYThis article has a reply.
VIEW THE REPLYPublication: Journal of Engineering Mechanics
Volume 133, Issue 7
Abstract
Mathematical models are developed and used to study the properties of complex systems in just about every area of applied science and engineering. Information on the system being modeled is, in general, incomplete, so that there may be two or more models consistent with the available information. The collection of these models is called the class of candidate models. A decision-theoretic method is developed for selecting the optimal member from the collection. The optimal model depends on the available information, the class of candidate models, and the model use. The candidate models may be deterministic or random. Classical methods for model selection, including the method of maximum likelihood and Bayesian methods, are briefly reviewed. These methods ignore model use and require data to be available. In addition, examples are used to show that classical methods for model selection can be unreliable in the sense that they can deliver unsatisfactory models when data is limited. The proposed decision-theoretic method for model selection does not have these limitations. The method accounts for model use via a utility function. This feature is especially important when modeling high-risk systems where the consequences of using an inappropriate model for the system can be disastrous.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
This work was supported by Sandia National Laboratories. Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company, for the United States Department of Energy’s National Nuclear Security Administration under Contract No. DE-AC04-94AL85000.
References
Abramowitz, M., and Stegun, I. A. (1972). Handbook of mathematical functions, Dover, New York.
Alvin, K. F., Oberkampf, W. L., Rutherford, B. M., and Diegert, K. V. (2000). “Methodology for characterizing modeling and discretization uncertainties in computational simulation.” Rep. No. SAND 2000-0515, Sandia National Laboratories, Albuquerque, N.M., Appendix D.
Atkinson, S., Woods, R., and Sivapalan, M. (2002). “Climate and landscape controls on water balance model complexity over changing timescales.” Water Resour. Res., 38(12), 1314.
Beck, J. L., and Katafygiotis, L. S. (1998). “Updating models and their uncertainties. Part I: Bayesian statistical framework.” J. Eng. Mech. 124(4), 455–461.
Beck, J. L., and Yuen, K.-V. (2004). “Model selection using response measurements: Bayesian probabilistic approach.” J. Eng. Mech., 130(2), 192–203.
Blackwell, D., and Girshick, M. A. (1954). Theory of games and statistical decisions, Dover, New York.
Brockwell, P. J., and Davis, R. A. (1991). Time series: Theory and methods, Springer, New York.
Burnham, K. P., and Anderson, D. R. (1998). Model selection and inference: A practical information-theoretic approach, Springer, New York.
Chernoff, H., and Moses, L. E. (1959). Elementary decision theory, Dover, New York.
Christoffersen, P. F., and Diebold, F. X. (1996). “Further results on forecasting and model selection under asymmetric loss.” J. Appl. Econ., 11(5), 561–571.
Cressie, N. A. C. (1993). Statistics for spatial data, Wiley Series in Probability and Mathematical Statistics, Wiley, New York.
Ditlevsen, O., and Madsen, H. O. (2005). Structural reliability methods, Ed. 2.2.5: Internet Ed., July 2005, ⟨http://www.web.mek.dtu.dk/staff/od/books./OD-HOM-StrucRelMeth-Ed2.2.5.pdf⟩ (April 19, 2007).
Feller, W. (1968). An introduction to probability theory and its applications, Vol. I, 3rd Ed., Wiley, New York.
Field, R. V., Jr. (2004). “Methods for model selection in applied science and engineering.” Ph.D. thesis, Cornell Univ., Ithaca, N.Y.
Field, R. V., Jr., and Grigoriu, M. (2006a). “Optimal design of sensor networks for vehicle detection, classification, and monitoring.” Probab. Eng. Mech., 21(4), 305–316.
Field, R. V., Jr., and Grigoriu, M. (2006b). “Optimal stochastic models for spacecraft atmospheric re-entry.” J. Sound Vib., 290(3–5), 991–1014.
Geyskens, G., and Der Kiureghian, A. (1996). “Bayesian assessment and selection of models for structural reliability analysis.” 7th Specialty Conf. on Probabilistic Mechanics and Structural and Geotechnical Reliability, Worcester, Mass., 566–569.
Grigoriu, M. (1976). “A decision theoretic approach to model selection for structural reliability.” Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, Mass.
Grigoriu, M. (1984). “Crossings of non-Gaussian translation processes.” J. Eng. Mech., 110(4), 610–620.
Grigoriu, M. (2002). Stochastic calculus: Applications in science and engineering, Birkhäuser, Boston.
Grigoriu, M., Veneziano, D., and Cornell, C. A. (1979). “Probabilistic modelling as decision making.” J. Engrg. Mech. Div., 105(4), 585–596.
Hjelmstad, K. D. (1997). Fundamentals of structural mechanics, Prentice-Hall, Upper Saddle River, N.J.
Hjorth, J. S. U. (1994). Computer intensive statistical methods: Validation, model, selection, and bootstrap, Chapman & Hall, London.
Igusa, T., Buonopane, S., and Ellingwood, B. (2002). “Bayesian analysis of uncertainty for structural engineering applications.” Struct. Safety, 24(2–4), 165–186.
Katafygiotis, L. S., and Beck, J. L. (1998). “Updating models and their uncertainties. Part II: Model identifiability.” J. Eng. Mech., 124(4), 463–467.
Keeping, E. S. (1995). Introduction to statistical inference, Dover, New York.
Martijn Cremers, K. J. (2002). “Stock return predictability: A Bayesian model selection perspective.” Rev. Financ. Stud., 15(4), 1223–1249.
Mitchell, H., and McKenzie, M. D. (2003). “GARCH model selection criteria.” Quant. Finance, 3(4), 262–284.
Papadimitriou, C., Beck, J., and Katafygiotis, L. (2001). “Updating robust reliability using structural test data.” Probab. Eng. Mech., 16(2), 103–113.
Raiffa, H., and Schlaifer, R. (1961). Applied statistical decision theory, Div. of Research, Harvard Business School, Cambridge, Mass.
Reese, G., Bhardwaj, M., Segalman, D., Alvin, K., and Driessen, B. (1999). “Salinas: User’s notes.” Rep. No. SAND99-2801, Sandia National Laboratories, Albuquerque, N.M.
Robert, C. P. (2001). The Bayesian choice, Springer, New York.
Soong, T. T., and Grigoriu, M. (1993). Random vibration of mechanical and structural systems, Prentice-Hall, Englewood Cliffs, N.J.
van Gelder, P., van Noortwijk, J., and Duits, M. (1999). “Selection of probability distributions with a case study on extreme Oder River discharges.” Proc., 10th European Safety and Reliability Conf., Balkema, Rotterdam, The Netherlands, 1475–1480.
van Noortwijk, J. M., Kalk, H. J., Duits, M. T., and Chbab E. H. (2001). “The use of Bayes factors for model selection in structural reliability.” Structural Safety and Reliability: Proc., 8th Int. Conf. on Structural Safety and Reliability (ICOSSAR), R. B. Corotis, G. I. Schuëller, and M. Shinozuka, eds., Balkema, Lisse.
Vehtari, A. (2001). “Bayesian model assessment and selection using expected utilities.” Ph.D. thesis, Helsinki Univ. of Technology, Finland.
von Neumann, J., and Morgenstern, O. (1944). Theory of games and economical behavior, Princeton University Press, Princeton, N.J.
Wood, E. (1974). “The methodology of Bayesian inference and decision making applied to extreme hydrological events.” Ph.D. thesis, Massachusetts Institute of Technology, Cambridge, Mass.
Yuen, K.-V., and Beck, J. (2003). “Updating properties of nonlinear dynamical systems with uncertain input.” J. Eng. Mech., 129(1), 9–20.
Zellner, A. (1971). An introduction to Bayesian inference in econometrics, Wiley, New York.
Zhang, R., and Mahadevan, S. (2000). “Model uncertainty and Bayesian updating in reliability-based inspection.” Struct. Safety, 22(2), 145–160.
Information & Authors
Information
Published In
Journal of Engineering Mechanics
Volume 133 • Issue 7 • July 2007
Pages: 780 - 791
Copyright
© 2007 ASCE.
History
Received: Oct 25, 2004
Accepted: Dec 14, 2006
Published online: Jul 1, 2007
Published in print: Jul 2007
Notes
Note. Associate Editor: Arvid Naess
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.