Abstract
Models are used to represent and characterize physical phenomena. When there are many plausible models for a particular phenomenon, the modeler can exploit the computational tool called model falsification to systematically eliminate models that do not reasonably fit measured data. Model falsification typically compares measurements and their predictions by different models, and rejects a model if some metric of the difference between them is outside some prescribed bounds. This paper compares two model falsification approaches: a conventional bounds on residual errors and a proposed bounds on a model’s prediction of the likelihood of the residual errors. The bounds in both approaches are selected based on two error control criteria: the more commonly used familywise error rate (FWER) and—proposed herein for model falsification—the false discovery rate (FDR). Because FDR control significantly increases the likelihood of rejecting an invalid model when there are many measurements, FDR provides advantages over FWER in exploratory studies. A variant of the second approach, using likelihood bounds specified by a constant probability mass contained within those bounds, is also investigated. Unlike many model falsification studies, the focus herein is on systems with many measurements, spread across spatial and/or temporal dimensions, such as dynamical systems. An elementary example is used to show the principles of each approach. A second example considers a series of four-degree-of-freedom models of a structure subjected to an earthquake excitation. The results from these examples show that FDR does indeed increase the number of falsified models, whereas the use of likelihood bounds additionally gives unfalsified models confidence values, which can also be used for maximum likelihood parameter estimation.
Get full access to this article
View all available purchase options and get full access to this article.
Acknowledgments
The authors gratefully acknowledge the partial support of this work by the National Science Foundation through awards CMMI 13-44937, 14-36018/14-36058, and 16-63667/16-62992. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation. Mr. De and Dr. Brewick acknowledge the support of a Viterbi Ph.D. Fellowship and a Viterbi Postdoctoral Fellowship, respectively, from University of Southern California.
References
Alberto, L.-G. (1994). Probability and random processes for electrical engineering, Addison-Wesley, Reading, MA.
Beck, J. L., and Taflanidis, A. A. (2013). “Prior and posterior robust stochastic predictions for dynamical systems using probability logic.” Int. J. Uncertainty Quantif., 3(4), 271–288.
Benjamini, Y., Heller, R., and Yekutieli, D. (2009). “Selective inference in complex research.” Philos. Trans. R. Soc. London, Ser. A, 367(1906), 4255–4271.
Benjamini, Y., and Hochberg, Y. (1995). “Controlling the false discovery rate: A practical and powerful approach to multiple testing.” J. R. Stat. Soc. Ser. B Methodol., 57(1), 289–300.
Benjamini, Y., and Yekutieli, D. (2001). “The control of the false discovery rate in multiple testing under dependency.” Ann. Stat., 29(4), 1165–1188.
Beven, K. (1993). “Prophecy, reality and uncertainty in distributed hydrological modelling.” Adv. Water Resour., 16(1), 41–51.
Beven, K., and Binley, A. (1992). “The future of distributed models: Model calibration and uncertainty prediction.” Hydrol. Process., 6(3), 279–298.
Beven, K., and Freer, J. (2001). “Equifinality, data assimilation, and uncertainty estimation in mechanistic modelling of complex environmental systems using the GLUE methodology.” J. Hydrol., 249(1), 11–29.
Beven, K. J. (2011). Rainfall-runoff modelling: The primer, Wiley, Hoboken, NJ.
Bianchi, F. D., and Sánchez-Peña, R. S. (2010). “Robust identification/invalidation in an LPV framework.” Int. J. Robust Nonlinear Control, 20(3), 301–312.
Bouaziz, M., Jeanmougin, M., and Guedj, M. (2012). “Multiple testing in large-scale genetic studies.” Data production and analysis in population genomics, Springer, New York, 213–233.
Box, G. E. P., and Draper, N. R. (1987). Empirical model-building and response surfaces, Wiley, New York.
Brugarolas, P. B., and Safonov, M. G. (2004). “Learning about dynamical systems via unfalsification of hypotheses.” Int. J. Robust Nonlinear Control, 14(11), 933–943.
Dunn, O. J. (1961). “Multiple comparisons among means.” J. Am. Stat. Assoc., 56(293), 52–64.
Goulet, J.-A., Coutu, S., and Smith, I. F. C. (2013a). “Model falsification diagnosis and sensor placement for leak detection in pressurized pipe networks.” Adv. Eng. Inf., 27(2), 261–269.
Goulet, J.-A., Kripakaran, P., and Smith, I. F. C. (2010). “Multimodel structural performance monitoring.” J. Struct. Eng., 1309–1318.
Goulet, J.-A., Michel, C., and Smith, I. F. C. (2013b). “Hybrid probabilities and error-domain structural identification using ambient vibration monitoring.” Mech. Syst. Signal Proc., 37(1–2), 199–212.
Goulet, J.-A., and Smith, I. F. C. (2013a). “Performance-driven measurement system design for structural identification.” J. Comput. Civ. Eng., 427–436.
Goulet, J.-A., and Smith, I. F. C. (2013b). “Predicting the usefulness of monitoring for identifying the behavior of structures.” J. Struct. Eng., 1716–1727.
Goulet, J.-A., and Smith, I. F. C. (2013c). “Structural identification with systematic errors and unknown uncertainty dependencies.” Comput. Struct., 128, 251–258.
Goulet, J.-A., Texier, M., Michel, C., Smith, I. F. C., and Chouinard, L. (2014). “Quantifying the effects of modeling simplifications for structural identification of bridges.” J. Bridge Eng., 59–71.
Hasenauer, J., Waldherr, S., Wagner, K., and Allgower, F. (2010). “Parameter identification, experimental design and model falsification for biological network models using semidefinite programming.” IET Syst. Biol., 4(2), 119–130.
Hoel, P. G., Port, S. C., and Stone, C. J. (1971). Introduction to statistical theory, Houghton Mifflin, Boston.
Hwang, J. S., and Chiou, J. M. (1996). “An equivalent linear model of lead-rubber seismic isolation bearings.” Eng. Struct., 18(7), 528–536.
Jaynes, E. T. (1957). “Information theory and statistical mechanics.” Phys. Rev., 106(4), 620–630.
Kawashima, K., Hasegawa, K., and Nagashima, H. (1992). “Manual for Menshin design of highway bridges.” Proc., 2nd US-Japan Workshop on Earthquake Protective Systems for Bridges, Public Works Research Institute, Tsukuba City, Japan.
Mahalanobis, P. C. (1936). “On the generalized distance in statistics.” Proc., National Institute of Sciences, Vol. 2, Calcutta, India, 49–55.
Mantovan, P., and Todini, E. (2006). “Hydrological forecasting uncertainty assessment: Incoherence of the GLUE methodology.” J. Hydrol., 330(1), 368–381.
McFarland, J., and Mahadevan, S. (2008). “Multivariate significance testing and model calibration under uncertainty.” Comput. Methods Appl. Mech. Eng., 197(29), 2467–2479.
Moser, A. (1995). “‘Macroscopic pattern analysis’ based on formal analogies as a scientific methodology for complex systems.” Acta Biotechnol., 15(2), 173–195.
Nagarajaiah, S., and Sun, X. (2000). “Response of base-isolated USC hospital building in Northridge earthquake.” J. Struct. Eng., 1177–1186.
Øksendal, B. (2013). Stochastic differential equations: An introduction with applications, Springer, Berlin.
Poolla, K., Khargonekar, P., Tikku, A., Krause, J., and Nagpal, K. (1994). “A time-domain approach to model validation.” IEEE Trans. Autom. Control, 39(5), 951–959.
Popper, K. R. (2002). The logic of scientific discovery, Routledge, New York. Translation of Logik der Forschung, Springer, Vienna, Austria, 1934.
Ramallo, J. C., Johnson, E. A., and Spencer, B. F., Jr. (2002). “‘Smart’ base isolation systems.” J. Eng. Mech., 1088–1099.
Raphael, B., and Smith, I. F. C. (1998). “Finding the right model for bridge diagnosis.” Artificial intelligence in structural engineering, Vol. 1454, Springer, Berlin, 308–319.
Robert-Nicoud, Y., Raphael, B., and Smith, I. F. C. (2005). “System identification through model composition and stochastic research.” J. Comput. Civ. Eng., 239–247.
Safonov, M. G., and Tsao, T. (1997). “The unfalsified control concept and learning.” IEEE Trans. Autom. Control, 42(6), 843–847.
Schweppe, F. C. (1968). “Recursive state estimation: Unknown but bounded errors and system inputs.” IEEE Trans. Autom. Control, 13(1), 22–28.
Šidák, Z. (1967). “Rectangular confidence regions for the means of multivariate normal distributions.” J. Am. Stat. Assoc., 62(318), 626–633.
Simes, R. J. (1986). “An improved Bonferroni procedure for multiple tests of significance.” Biometrika, 73(3), 751–754.
Smith, I. F. C., and Saitta, S. (2008). “Improving knowledge of structural system behavior through multiple models.” J. Struct. Eng., 553–561.
Smith, R., Dullerud, G., Rangan, S., and Poolla, K. (1997). “Model validation for dynamically uncertain systems.” Math. Modell. Syst. Methods Tools Appl. Eng. Relat. Sci., 3(1), 43–58.
Smith, R. S., and Doyle, J. C. (1992). “Model validation: A connection between robust control and identification.” IEEE Trans. Autom. Control, 37(7), 942–952.
Storey, J. D. (2003). “The positive false discovery rate: A Bayesian interpretation and the q-value.” Ann. Stat., 31(6), 2013–2035.
Sznaier, M., and Mazzaro, M. C. (2003). “An LMI approach to control-oriented identification and model (in)validation of LPV systems.” IEEE Trans. Autom. Control, 48(9), 1619–1624.
Tarantola, A. (2006). “Popper, Bayes and the inverse problem.” Nat. Phys., 2(8), 492–494.
van Ballmoos, P., Guessoum, N., Jean, P., and Knödlseder, J. (2003). “Models for the positive latitude annihilation feature.” Astron. Astrophys., 397(2), 635–643.
Wen, Y.-K. (1976). “Method for random vibration of hysteretic systems.” J. Eng. Mech. Div., 102(2), 249–263.
Zaki, M. J., and Meira, W., Jr. (2014). Data mining and analysis: Fundamental concepts and algorithms, Cambridge University Press, New York.
Information & Authors
Information
Published In
Journal of Engineering Mechanics
Volume 144 • Issue 9 • September 2018
Copyright
©2018 American Society of Civil Engineers.
History
Received: Jul 12, 2017
Accepted: Oct 19, 2017
Published online: Jun 25, 2018
Published in print: Sep 1, 2018
Discussion open until: Nov 25, 2018
ASCE Technical Topics:
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.