Computationally Efficient Adaptive Design of Experiments for Global Metamodeling through Integrated Error Approximation and Multicriteria Search Strategies
Publication: Journal of Engineering Mechanics
Volume 149, Issue 8
Abstract
Gaussian processes (GPs) are a popular technique for global metamodeling applications. Their objective in such settings is to establish an efficient and globally accurate approximation of the response surface of computationally expensive simulation models. When developing such GPs, the design of (simulation) experiments (DoE) plays an important role in reducing the required number of model runs for obtaining accurate approximations. Sequential (adaptive) selection of experiments can provide significant advantages, especially when the response surface is characterized by localized nonlinearities. Such adaptive DoE strategies for global metamodeling applications typically focus on minimizing the predictive GP variance, representing an exploration strategy, while recent developments have additionally considered the reduction of the GP bias obtained through cross validation, representing an exploitation strategy. While significant focus has been placed on the definition of appropriate adaptive DoE criteria, computational challenges still exist that limit the widespread adoption of adaptive DoE techniques—for example, related to the additional computational demand for identifying the optimal new experiment(s) or the necessity to establish proper schemes to combine exploration and exploitation strategies. To address these specific challenges, this research investigates two new adaptive DoE formulations. The first one focuses on the approximation of the popular integrated mean square error (IMSE) DoE criterion. The computationally demanding GP predictive variance update (after addition of each candidate experiment), required in the original IMSE formulation, is replaced by an approximation based on the current predictive variance and the domain of influence that surrounds each new experiment. The approximation is established through a parametric formulation that leverages the GP kernel to describe the aforementioned domain, with characteristics that are progressively calibrated across the GP training stages, to minimize the discrepancy between the actual and the approximated IMSE. The second formulation establishes a multicriteria search for simultaneously identifying multiple Pareto optimal experiments that balance exploration and exploitation objectives, replacing conventional strategies that establish a weighted combination of these objectives to promote a single DoE selection criterion.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
The DoE formulations discussed in this paper can be implemented through quoFEM (v3.2), an open-source research software for uncertainty quantification (UQ) in natural hazard engineering, developed by NHERI SimCenter (McKenna et al. 2022). Numerical codes for the DoE optimization can be obtained from the corresponding author upon reasonable request.
Acknowledgments
This research was financially supported by the National Science Foundation under Grant CMMI- 2131111. This support is gratefully acknowledged. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
References
Ankenman, B., B. L. Nelson, and J. Staum. 2008. “Stochastic Kriging for simulation metamodeling.” In Proc., 2008 Winter Simulation Conf., 362–370. New York: IEEE.
Asher, M. J., B. F. Croke, A. J. Jakeman, and L. J. Peeters. 2015. “A review of surrogate models and their application to groundwater modeling.” Water Resour. Res. 51 (8): 5957–5973. https://doi.org/10.1002/2015WR016967.
Beck, J., and S. Guillas. 2016. “Sequential design with mutual information for computer experiments (MICE): Emulation of a tsunami model.” SIAM/ASA J. Uncertainty Quantif. 4 (1): 739–766. https://doi.org/10.1137/140989613.
Beyer, K., J. Goldstein, R. Ramakrishnan, and U. Shaft. 1999. “When is nearest neighbor meaningful?” In Proc., Int. Conf. on Database Theory, 217–235. Berlin: Springer.
Blatman, G., and B. Sudret. 2010. “An adaptive algorithm to build up sparse polynomial chaos expansions for stochastic finite element analysis.” Probab. Eng. Mech. 25 (2): 183–197. https://doi.org/10.1016/j.probengmech.2009.10.003.
Bodenmann, L., Y. Reuland, and B. Stojadinovic. 2021. “Dynamic updating of building loss predictions using regional risk models and conventional post-earthquake data sources.” In Proc., 31st European Safety and Reliability Conf. (ESREL 2021). Singapore: Research Publishing Services.
Byun, J., and J. Song. 2021. “Generalized matrix-based Bayesian network for multi-state systems.” Reliab. Eng. Syst. Saf. 211 (Jul): 107468. https://doi.org/10.1016/j.ress.2021.107468.
Cole, D. A., R. B. Christianson, and R. B. Gramacy. 2021. “Locally induced Gaussian processes for large-scale simulation experiments.” Stat. Comput. 31 (3): 1–21. https://doi.org/10.1007/s11222-021-10007-9.
Contreras, M. T., J. Gironás, and C. Escauriaza. 2020. “Forecasting flood hazards in real time: A surrogate model for hydrometeorological events in an Andean watershed.” Nat. Hazards Earth Syst. Sci. 20 (12): 3261–3277. https://doi.org/10.5194/nhess-20-3261-2020.
Csató, L., and M. Opper. 2002. “Sparse on-line Gaussian processes.” Neural Comput. 14 (3): 641–668. https://doi.org/10.1162/089976602317250933.
Deb, K., A. Pratap, S. Agarwal, and T. Meyarivan. 2002. “A fast and elitist multiobjective genetic algorithm: NSGA-II.” IEEE Trans. Evol. Comput. 6 (2): 182–197. https://doi.org/10.1109/4235.996017.
Deierlein, G. G., and A. Zsarnóczay. 2019. State-of-art in computational simulation for natural hazards engineering. Zenodo. https://doi.org/10.5281/zenodo.4558106.
Deisenroth, M., and J. W. Ng. 2015. “Distributed Gaussian processes.” In Vol. 37 of Proc., 32nd Int. Conf. on Machine Learning, 1481–1490. Cambridge, MA: Journal of Machine Learning Research.
Dixon, L., and G. P. Szegö. 1978. Towards global optimisation. V2 ed. Amsterdam, Netherlands: North-Holland.
Dongarra, J. J., C. B. Moler, J. R. Bunch, and G. W. Stewart. 1979. LINPACK users’ guide. Philadelphia: SIAM, Society for Industrial and Applied Mathematics.
Dubrule, O. 1983. “Cross validation of Kriging in a unique neighborhood.” J. Int. Assoc. Math. Geol. 15 (6): 687–699. https://doi.org/10.1007/BF01033232.
Forrester, A. I., and A. J. Keane. 2009. “Recent advances in surrogate-based optimization.” Prog. Aerosp. Sci. 45 (1–3): 50–79. https://doi.org/10.1016/j.paerosci.2008.11.001.
Fuhg, J. N., A. Fau, and U. Nackenhorst. 2021. “State-of-the-art and comparative review of adaptive sampling methods for Kriging.” Arch. Comput. Methods Eng. 28 (4): 2689–2747. https://doi.org/10.1007/s11831-020-09474-6.
Garud, S. S., I. A. Karimi, and M. Kraft. 2017. “Design of computer experiments: A review.” Comput. Chem. Eng. 106 (Nov): 71–95. https://doi.org/10.1016/j.compchemeng.2017.05.010.
Ginsbourger, D. 2014. “Sequential design of computer experiments.” In Wiley StatsRef: Statistics reference online, 1–9. Hoboken, NJ: Wiley.
Gramacy, R. B. 2020. Surrogates: Gaussian process modeling, design, and optimization for the applied sciences. 1st ed. New York: Chapman and Hall/CRC.
Gunantara, N. 2018. “A review of multi-objective optimization: Methods and its applications.” Cogent Eng. 5 (1): 1502242. https://doi.org/10.1080/23311916.2018.1502242.
Jain, A. K., A. J. Mao, and K. M. Mohiuddin. 1996. “Artificial neural networks: A tutorial.” Computer 29 (3): 31–44. https://doi.org/10.1109/2.485891.
Jia, G., and A. A. Taflanidis. 2013. “Kriging metamodeling for approximation of high-dimensional wave and surge responses in real-time storm/hurricane risk assessment.” Comput. Methods Appl. Mech. Eng. 261 (Jul): 24–38. https://doi.org/10.1016/j.cma.2013.03.012.
Jia, G., and A. A. Taflanidis. 2014. “Sample-based evaluation of global probabilistic sensitivity measures.” Comput. Struct. 144 (Nov): 103–118. https://doi.org/10.1016/j.compstruc.2014.07.019.
Jiang, P., L. Shu, Q. Zhou, H. Zhou, X. Shao, and J. Xu. 2015. “A novel sequential exploration-exploitation sampling strategy for global metamodeling.” IFAC-PapersOnLine 48 (28): 532–537. https://doi.org/10.1016/j.ifacol.2015.12.183.
Jin, R., W. Chen, and A. Sudjianto. 2002. “On sequential sampling for global metamodeling in engineering design.” In Vol. 36223 of Proc., of the ASME 2002 Int. Design Engineering Technical Conf. and Computers and Information in Engineering Conf. Volume 2: 28th Design Automation Conf., 539–548. New York: ASME.
Johnson, M. E., L. M. Moore, and D. Ylvisaker. 1990. “Minimax and maximin distance designs.” J. Stat. Plann. Inference 26 (2): 131–148. https://doi.org/10.1016/0378-3758(90)90122-B.
Kanai, K. 1957. “Semi-empirical formula for the seismic characteristics of the ground.” Bull. Earthquake Res. Inst. Univ. Tokyo 35 (2): 309–325.
Kim, I. Y., and O. L. De Weck. 2005. “Adaptive weighted-sum method for bi-objective optimization: Pareto front generation.” Struct. Multidiscip. Optim. 29 (2): 149–158. https://doi.org/10.1007/s00158-004-0465-1.
Kim, J., and J. Song. 2020. “Probability-adaptive Kriging in n-Ball (PAK-Bn) for reliability analysis.” Struct. Saf. 85 (Jul): 101924. https://doi.org/10.1016/j.strusafe.2020.101924.
Kleijnen, J. P. 2009. “Kriging metamodeling in simulation: A review.” Eur. J. Oper. Res. 192 (3): 707–716. https://doi.org/10.1016/j.ejor.2007.10.013.
Kleijnen, J. P. 2017. “Regression and Kriging metamodels with their experimental designs in simulation: A review.” Eur. J. Oper. Res. 256 (1): 1–16. https://doi.org/10.1016/j.ejor.2016.06.041.
Kleijnen, J. P., and W. V. Beers. 2004. “Application-driven sequential designs for simulation experiments: Kriging metamodelling.” J. Oper. Res. Soc. 55 (8): 876–883. https://doi.org/10.1057/palgrave.jors.2601747.
Kleijnen, J. P., and W. C. Van Beers. 2022. “Statistical tests for cross-validation of Kriging models.” INFORMS J. Comput. 34 (1): 607–621. https://doi.org/10.1287/ijoc.2021.1072.
Koehler, J. R., and A. B. Owen. 1996. “9 Computer experiments.” In Vol. 13 of Handbook of statistics, edited by S. Ghosh and C. R. Rao, 261–308. Amsterdam, Netherlands: Elsevier Science and Technology.
Krause, A., A. Singh, and C. Guestrin. 2008. “Near-optimal sensor placements in Gaussian processes: Theory, efficient algorithms and empirical studies.” J. Mach. Learn. Res. 9 (2): 235–284.
Kyprioti, A. P., A. A. Taflanidis, N. C. Nadal-Caraballo, and M. Campbell. 2021. “Storm hazard analysis over extended geospatial grids utilizing surrogate models.” Coastal Eng. 168 (Sep): 103855. https://doi.org/10.1016/j.coastaleng.2021.103855.
Kyprioti, A. P., J. Zhang, and A. A. Taflanidis. 2020. “Adaptive design of experiments for global Kriging metamodeling through cross-validation information.” Struct. Multidiscip. Optim. 62 (3): 1135–1157. https://doi.org/10.1007/s00158-020-02543-1.
Le Gratiet, L., and C. Cannamela. 2015. “Cokriging-based sequential design strategies using fast cross-validation techniques for multi-fidelity computer codes.” Technometrics 57 (3): 418–427. https://doi.org/10.1080/00401706.2014.928233.
Liu, H., J. Cai, and Y. Ong. 2017. “An adaptive sampling approach for Kriging metamodeling by maximizing expected prediction error.” Comput. Chem. Eng. 106 (Nov): 171–182. https://doi.org/10.1016/j.compchemeng.2017.05.025.
Liu, H., Y. Ong, and J. Cai. 2018. “A survey of adaptive sampling for global metamodeling in support of simulation-based complex engineering design.” Struct. Multidiscip. Optim. 57 (1): 393–416. https://doi.org/10.1007/s00158-017-1739-8.
Liu, H., Y. Ong, X. Shen, and J. Cai. 2020. “When Gaussian process meets big data: A review of scalable GPs.” IEEE Trans. Neural Networks Learn. Syst. 31 (11): 4405–4423. https://doi.org/10.1109/TNNLS.2019.2957109.
Liu, H., S. Xu, Y. Ma, X. Chen, and X. Wang. 2016. “An adaptive Bayesian sequential sampling approach for global metamodeling.” J. Mech. Des. 138 (1): 011404. https://doi.org/10.1115/1.4031905.
Lophaven, S. N., H. B. Nielsen, and J. Søndergaard. 2002. Vol. 2 of DACE: A Matlab kriging toolbox. Lyngby, Denmark: IMM, Informatics and Mathematical Modelling.
Lutes, L. D., and S. Sarkani. 2004. Random Vibrations. Oxford, UK: Elsevier Science and Technology.
McBride, K., and K. Sundmacher. 2019. “Overview of surrogate modeling in chemical process engineering.” Chem. Ing. Tech. 91 (3): 228–239. https://doi.org/10.1002/cite.201800091.
McKay, M. D., R. J. Beckman, and W. J. Conover. 2000. “A comparison of three methods for selecting values of input variables in the analysis of output from a computer code.” Technometrics 42 (1): 55–61. https://doi.org/10.1080/00401706.2000.10485979.
McKenna, F., S. Yi, B. S. Aakash, A. Zsarnoczay, M. Gardner, and W. Elhaddad. 2022. NHERI-SimCenter/quoFEM: Version 3.2.0 (v3.2.0). Zenodo. https://doi.org/10.5281/zenodo.7131444.
Moustapha, M., S. Marelli, and B. Sudret. 2022. “Active learning for structural reliability: Survey, general framework and benchmark.” Struct. Saf. 96 (2): 102174. https://doi.org/10.1016/j.strusafe.2021.102174.
Page, T. J. 1984. “Multivariate statistics: A vector space approach.” J. Mark. Res. 21 (2): 236. https://doi.org/10.1177/002224378402100214.
Pandita, P., P. Tsilifis, N. M. Awalgaonkar, I. Bilionis, and J. Panchal. 2021. “Surrogate-based sequential Bayesian experimental design using non-stationary Gaussian processes.” Comput. Methods Appl. Mech. Eng. 385 (Nov): 114007. https://doi.org/10.1016/j.cma.2021.114007.
Picheny, V., D. Ginsbourger, O. Roustant, R. T. Haftka, and N.-H. Kim. 2010. “Adaptive designs of experiments for accurate approximation of a target region.” J. Mech. Des. 132 (7): 071008. https://doi.org/10.1115/1.4001873.
Prebeg, P., V. Zanic, and B. Vazic. 2014. “Application of a surrogate modeling to the ship structural design.” Ocean Eng. 84 (Jul): 259–272. https://doi.org/10.1016/j.oceaneng.2014.03.032.
Provost, F., D. Jensen, and T. Oates. 1999. “Efficient progressive sampling.” In Proc., Conf. on Knowledge Discovery in Data: Proc. of the Fifth ACM SIGKDD Int. Conf. on Knowledge Discovery and Data Mining, 23–32. New York: Association for Computing Machinery.
Rasmussen, C. E., and C. K. Williams. 2006. Gaussian processes for machine learning. Cambridge, MA: MIT Press.
Razavi, S., B. A. Tolson, and D. H. Burn. 2012. “Review of surrogate modeling in water resources.” Water Resour. Res. 48 (7): W07401. https://doi.org/10.1029/2011WR011527.
Roustant, O., D. Ginsbourger, and Y. Deville. 2012. “DiceKriging, DiceOptim: Two R packages for the analysis of computer experiments by Kriging-based metamodelling and optimization.” J. Stat. Software 51 (Oct): 1–55. https://doi.org/10.18637/jss.v051.i01.
Sacks, J., S. B. Schiller, and W. J. Welch. 1989a. “Designs for computer experiments.” Technometrics 31 (1): 41–47. https://doi.org/10.1080/00401706.1989.10488474.
Sacks, J., W. J. Welch, T. J. Mitchell, and H. P. Wynn. 1989b. “Design and analysis of computer experiments.” Stat. Sci. 4 (4): 409–423. https://doi.org/10.1214/ss/1177012413.
Stewart, G. W. 1998. Matrix algorithms. Philadelphia: Society for Industrial and Applied Mathematics.
Strang, G. 2016. Introduction to linear algebra. 5th ed. Wellesley, MA: Wellesley-Cambridge Press.
Sundararajan, S., and S. S. Keerthi. 2001. “Predictive approaches for choosing hyperparameters in Gaussian processes.” Neural Comput. 13 (5): 1103–1118. https://doi.org/10.1162/08997660151134343.
Tajimi, H. 1960. “Statistical method of determining the maximum response of building structure during an earthquake.” In Proc., 2nd WCEE 2, 781–798. Tokyo: Science Council of Japan.
Vazquez, E., and J. Bect. 2011. “Sequential search based on Kriging: Convergence analysis of some algorithms.” Preprint, submitted November 16, 2011. https://arxiv.org/abs/1111.3866.
Welch, W. J. 1983. “A mean squared error criterion for the design of experiments.” Biometrika 70 (1): 205–213. https://doi.org/10.1093/biomet/70.1.205.
Xu, S., H. Liu, X. Wang, and X. Jiang. 2014. “A robust error-pursuing sequential sampling approach for global metamodeling based on voronoi diagram and cross validation.” J. Mech. Des. 136 (7): 071009. https://doi.org/10.1115/1.4027161.
Zhang, J., and A. A. Taflanidis. 2018. “Adaptive Kriging stochastic sampling and density approximation and its application to rare-event estimation.” ASCE-ASME J. Risk Uncertainty Eng. Syst. Part A: Civ. Eng. 4 (3): 04018021. https://doi.org/10.1061/AJRUA6.0000969.
Zhang, J., A. A. Taflanidis, N. C. Nadal-Caraballo, J. A. Melby, and F. Diop. 2018. “Advances in surrogate modeling for storm surge prediction: Storm selection and addressing characteristics related to climate change.” Nat. Hazards 94 (3): 1225–1253. https://doi.org/10.1007/s11069-018-3470-1.
Information & Authors
Information
Published In
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Nov 23, 2022
Accepted: Feb 3, 2023
Published online: May 31, 2023
Published in print: Aug 1, 2023
Discussion open until: Oct 31, 2023
ASCE Technical Topics:
- Adaptive systems
- Approximation methods
- Computer models
- Engineering fundamentals
- Errors (statistics)
- Gaussian process
- Mathematics
- Methodology (by type)
- Model accuracy
- Models (by type)
- Probability
- Research methods (by type)
- Simulation models
- Statistics
- Stochastic processes
- Systems engineering
- Systems management
- Validation
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.