Efficient Data-Driven Modeling of Nonlinear Dynamical Systems via Metalearning
Publication: Journal of Engineering Mechanics
Volume 149, Issue 3
Abstract
Data-driven modeling of nonlinear dynamical systems is essential because of the need for a trade-off among complexity, efficiency, and reliability in analytical or numerical studies as well as the difficulty of deriving fully physics-based models. An important limitation is commonly seen in existing works: modeling of a new dynamical system typically starts from scratch, requiring a large amount of data and intensive computation, though some prior experience or knowledge is available from a previously collected database of similar but different systems. However, on the one hand, the data amount for the new dynamical system is often limited, especially for real-world dynamical systems. On the other hand, the computational resource is also limited and a data-driven modeling task is usually computationally expensive, especially for large-scale systems. To improve data efficiency and computational efficiency in data-driven modeling of nonlinear systems, we present an enhanced data-driven modeling approach by incorporating metalearning into a physics-integrated deep learning framework. The core idea is to learn the metaknowledge about how to model a new system from a previously collected database of similar but different systems. Then this metaknowledge is leveraged to enable efficient modeling of a new system with limited data. For validations we conducted numerical experiments on three sets of fundamental nonlinear systems, including Duffing oscillators, nonlinear pendulums, and van der Pol oscillators. We performed both interpolation and extrapolation modelings to investigate the generalization ability of the presented approach. Furthermore, we conducted a quantitative analysis on data efficiency, addressing two critical issues: how few data are sufficient for the new system modeling and how much prior experience (previously collected database of similar but different systems) is needed for the metalearning. The results show that the presented approach improves both data efficiency and computational efficiency, compared with the conventional data-driven modeling approach (without leverage of the prior database) and the pretraining-based approach (simply using the prior database but without metalearning idea). We also discuss the limitations of this work and potential future study.
Get full access to this article
View all available purchase options and get full access to this article.
Data Availability Statement
All data, models, or code that support the findings of this study are available from the corresponding author upon reasonable request.
Acknowledgments
This research was partially funded by the Physics of Artificial Intelligence Program of US Defense Advanced Research Projects Agency (DARPA) and the Michigan Technological University faculty startup fund.
References
Andrychowicz, M., M. Denil, S. G. Colmenarejo, M. W. Hoffman, D. Pfau, T. Schaul, B. Shillingford, and N. de Freitas. 2016. “Learning to learn by gradient descent by gradient descent.” In Proc., 30th Int. Conf. on Neural Information Processing Systems, 3988–3996. New York: Curran Associates.
Åström, K. J., and K. Furuta. 2000. “Swinging up a pendulum by energy control.” Automatica 36 (2): 287–295. https://doi.org/10.1016/S0005-1098(99)00140-5.
Baek, J., D. B. Lee, and S. J. Hwang. 2020. “Learning to extrapolate knowledge: Transductive few-shot out-of-graph link prediction.” In Proc., Advances in Neural Information Processing Systems 33: Annual Conf. on Neural Information Processing Systems 2020. Red Hook, NY: Curran Associates.
Banerjee, S., J. Harrison, P. M. Furlong, and M. Pavone. 2020. Adaptive meta-learning for identification of rover-terrain dynamics. New York: Association for Computing Machinery.
Bengio, S., Y. Bengio, J. Cloutier, and J. Gescei. 2013. “On the optimization of a synaptic learning rule.” In Optimality in biological and artificial networks, 281–303. New York: Routledge.
Biggs, J. B. 1985. “The role of metalearning in study processes.” Br. J. Educ. Psychol. 55 (3): 185–212. https://doi.org/10.1111/j.2044-8279.1985.tb02625.x.
Brunton, S. L., J. L. Proctor, J. N. Kutz, and W. Bialek. 2016. “Discovering governing equations from data by sparse identification of nonlinear dynamical systems.” Proc. Natl. Acad. Sci. U.S.A. 113 (15): 3932–3937. https://doi.org/10.1073/pnas.1517384113.
Cenedese, M., J. Axås, B. Bäuerlein, K. Avila, and G. Haller. 2022. “Data-driven modeling and prediction of non-linearizable dynamics via spectral submanifolds.” Nat. Commun. 13 (1): 872. https://doi.org/10.1038/s41467-022-28518-y.
Champion, K., B. Lusch, J. Nathan Kutz, and S. L. Brunton. 2019. “Data-driven discovery of coordinates and governing equations.” Proc. Natl. Acad. Sci. U.S.A. 116 (45): 22445–22451. https://doi.org/10.1073/pnas.1906995116.
Chen, Z., Y. Fu, Y.-X. Wang, L. Ma, W. Liu, and M. Hebert. 2019. “Image deformation meta-networks for one-shot learning.” In Proc., IEEE/CVF Conf. on Computer Vision and Pattern Recognition (CVPR). New York: IEEE.
Chua, A. J., C. R. Galley, and M. Vallisneri. 2019. “Reduced-order modeling with artificial neurons for gravitational-wave inference.” Phys. Rev. Lett. 122 (21): 211101. https://doi.org/10.1103/PhysRevLett.122.211101.
Cui, T., Y. M. Marzouk, and K. E. Willcox. 2015. “Data-driven model reduction for the Bayesian solution of inverse problems.” Int. J. Numer. Methods Eng. 102 (5): 966–990. https://doi.org/10.1002/nme.4748.
Ehsan, F., and R. H. Scanlan. 1990. “Vortex-induced vibrations of flexible bridges.” J. Eng. Mech. 116 (6): 1392–1411. https://doi.org/10.1061/(ASCE)0733-9399(1990)116:6(1392).
Finn, C., P. Abbeel, and S. Levine. 2017a. “Model-agnostic meta-learning for fast adaptation of deep networks.” In Proc., 34th Int. Conf. on Machine Learning, ICML 2017, 1856–1868. Brookline, MA: Microtome Publishing.
Finn, C., and S. Levine. 2018. “Meta-learning and universality: Deep representations and gradient descent can approximate any learning algorithm.” In Proc., Int. Conf. on Learning Representations. Washington, DC: OpenReview.net.
Finn, C., T. Yu, T. Zhang, P. Abbeel, and S. Levine. 2017b. “One-shot visual imitation learning via meta-learning.” In Proc., 1st Annual Conf. on Robot Learning. Brookline, MA: Microtome Publishing.
Gupta, R., and R. Jaiman. 2022. “Three-dimensional deep learning-based reduced order model for unsteady flow dynamics with variable Reynolds number.” Phys. Fluids 34 (3): 82741. https://doi.org/10.1063/5.0082741.
Harrison, J., A. Sharma, and M. Pavone. 2020. “Meta-learning priors for efficient online Bayesian regression.” In Proc., Algorithmic Foundations of Robotics XIII, 318–337. Berlin: Springer.
Hesthaven, J. S., and S. Ubbiali. 2018. “Non-intrusive reduced order modeling of nonlinear problems using neural networks.” J. Comput. Phys. 363 (Jun): 55–78. https://doi.org/10.1016/j.jcp.2018.02.037.
Hospedales, T., A. Antoniou, P. Micaelli, and A. Storkey. 2022. “Meta-learning in neural networks: A survey.” IEEE Trans. Pattern Anal. Mach. Intell. 44 (9): 5149–5169. https://doi.org/10.1109/TPAMI.2021.3079209.
Jin, X., P. Cheng, W. L. Chen, and H. Li. 2018. “Prediction model of velocity field around circular cylinder over various Reynolds numbers by fusion convolutional neural networks based on pressure on the cylinder.” Phys. Fluids 30 (4): 047105. https://doi.org/10.1063/1.5024595.
Karniadakis, G. E., I. G. Kevrekidis, L. Lu, P. Perdikaris, S. Wang, and L. Yang. 2021. “Physics-informed machine learning.” Nat. Rev. Phys. 3 (6): 422–440. https://doi.org/10.1038/s42254-021-00314-5.
Kerschen, G., K. Worden, A. F. Vakakis, and J. C. Golinval. 2006. “Past, present and future of nonlinear system identification in structural dynamics.” Mech. Syst. Sig. Process. 20 (3): 505–592. https://doi.org/10.1016/j.ymssp.2005.04.008.
Korda, M., and I. Mezić. 2018. “Linear predictors for nonlinear dynamical systems: Koopman operator meets model predictive control.” Automatica 93 (Mar): 149–160. https://doi.org/10.1016/j.automatica.2018.03.046.
Lake, B. M., R. Salakhutdinov, and J. B. Tenenbaum. 2015. “Human-level concept learning through probabilistic program induction.” Science 350 (6266): 1332–1338. https://doi.org/10.1126/science.aab3050.
Li, D., Y. Yang, Y. Z. Song, and T. M. Hospedales. 2018a. “Learning to generalize: Meta-learning for domain generalization.” In Proc., 32nd AAAI Conf. on Artificial Intelligence, 3490–3497. Washington, DC: Association for the Advancement of Artificial Intelligence Press.
Li, S., S. Laima, and H. Li. 2018b. “Data-driven modeling of vortex-induced vibration of a long-span suspension bridge using decision tree learning and support vector regression.” J. Wind Eng. Ind. Aerodyn. 172 (5): 196–211. https://doi.org/10.1016/j.jweia.2017.10.022.
Li, S., and Y. Yang. 2021. “Data-driven identification of nonlinear normal modes via physics-integrated deep learning.” Nonlinear Dyn. 106 (1): 3231–3246. https://doi.org/10.1007/s11071-021-06931-0.
Li, W., S. Laima, X. Jin, W. Yuan, and H. Li. 2020. “A novel long short-term memory neural-network-based self-excited force model of limit cycle oscillations of nonlinear flutter for various aerodynamic configurations.” Nonlinear Dyn. 100 (3): 2071–2087. https://doi.org/10.1007/s11071-020-05631-5.
Lusch, B., J. N. Kutz, and S. L. Brunton. 2018. “Deep learning for universal linear embeddings of nonlinear dynamics.” Nat. Commun. 9 (1): 4950. https://doi.org/10.1038/s41467-018-07210-0.
Noël, J. P., and G. Kerschen. 2017. “Nonlinear system identification in structural dynamics: 10 more years of progress.” Mech. Syst. Sig. Process. 83 (Jun): 2–35. https://doi.org/10.1016/j.ymssp.2016.07.020.
Pan, S., and K. Duraisamy. 2018. “Long-time predictive modeling of nonlinear dynamical systems using neural networks.” Complexity 2018 (1): 1–18. https://doi.org/10.1155/2018/4801012.
Patel, R. G., I. Manickam, N. A. Trask, M. A. Wood, M. Lee, I. Tomas, and E. C. Cyr. 2022. “Thermodynamically consistent physics-informed neural networks for hyperbolic systems.” J. Comput. Phys. 449 (Jan): 110754. https://doi.org/10.1016/j.jcp.2021.110754.
Pathak, J., B. Hunt, M. Girvan, Z. Lu, and E. Ott. 2018. “Model-free prediction of large spatiotemporally chaotic systems from data: A reservoir computing approach.” Phys. Rev. Lett. 120 (2): 24102. https://doi.org/10.1103/PhysRevLett.120.024102.
Quaranta, G., W. Lacarbonara, and S. F. Masri. 2020. “A review on computational intelligence for identification of nonlinear dynamical systems.” Nonlinear Dyn. 99 (2): 1709–1761. https://doi.org/10.1007/s11071-019-05430-7.
Raissi, M., P. Perdikaris, and G. E. Karniadakis. 2019. “Physics-informed neural networks: A deep learning framework for solving forward and inverse problems involving nonlinear partial differential equations.” J. Comput. Phys. 378 (Feb): 686–707. https://doi.org/10.1016/j.jcp.2018.10.045.
Ravi, S., and H. Larochelle. 2017. “Optimization as a model for few-shot learning.” In Proc., 5th Int. Conf. on Learning Representations. Washington, DC: OpenReview.net.
Rudy, S. H., S. L. Brunton, J. L. Proctor, and J. N. Kutz. 2017. “Data-driven discovery of partial differential equations.” Sci. Adv. 3 (4): e1602614. https://doi.org/10.1126/sciadv.1602614.
Rusu, A. A., D. Rao, J. Sygnowski, O. Vinyals, R. Pascanu, S. Osindero, and R. Hadsell. 2019. “Meta-learning with latent embedding optimization.” In Proc., 7th Int. Conf. on Learning Representations. Washington, DC: OpenReview.net.
Sæmundsson, S., K. Hofmann, and M. P. Deisenroth. 2018. “Meta reinforcement learning with latent variable Gaussian processes.” In Proc., 34th Conf. on Uncertainty in Artificial Intelligence 2018, 642–652. Arlington, VA: Association for Uncertainty in Artificial Intelligence.
Sahoo, D., Q. Pham, J. Lu, and S. C. Hoi. 2018. “Online deep learning: Learning deep neural networks on the fly.” In Proc., IJCAI Int. Joint Conf. on Artificial Intelligence, 2660–2666. Sacramento, CA: International Joint Conference on Artificial Intelligence.
Schweighofer, N., and K. Doya. 2003. “Meta-learning in reinforcement learning.” Neural Networks 16 (1): 5–9. https://doi.org/10.1016/S0893-6080(02)00228-9.
Simpson, T., N. Dervilis, and E. Chatzi. 2021. “Machine learning approach to model order reduction of nonlinear systems via autoencoder and LSTM networks.” J. Eng. Mech. 147 (10): 04021061. https://doi.org/10.1061/(ASCE)EM.1943-7889.0001971.
Teng, Q., and L. Zhang. 2019. “Data driven nonlinear dynamical systems identification using multi-step CLDNN.” AIP Adv. 9 (8): 085311. https://doi.org/10.1063/1.5100558.
Thrun, S., and L. Pratt. 1998. “Learning to learn: Introduction and overview.” In Learning to learn, 3–17. Berlin: Springer.
Trask, N., A. Huang, and X. Hu. 2022. “Enforcing exact physics in scientific machine learning: A data-driven exterior calculus on graphs.” J. Comput. Phys. 456 (5): 110969. https://doi.org/10.1016/j.jcp.2022.110969.
Wang, J. X., Z. Kurth-Nelson, D. Tirumala, H. Soyer, J. Z. Leibo, R. Munos, C. Blundell, D. Kumaran, and M. Botvinick. 2016. Learning to reinforcement learn. Washington, DC: OpenReview.net.
Wang, Y.-X., D. Ramanan, and M. Hebert. 2019. “Meta-learning to detect rare objects.” In Proc., IEEE/CVF Int. Conf. on Computer Vision (ICCV). New York: IEEE.
Williams, M. O., I. G. Kevrekidis, and C. W. Rowley. 2015. “A data-driven approximation of the Koopman operator: Extending dynamic mode decomposition.” J. Nonlinear Sci. 25 (6): 1307–1346. https://doi.org/10.1007/s00332-015-9258-5.
Zhou, F., B. Wu, and Z. Li. 2018. Deep meta-learning: Learning to learn in the concept space. New York: Association for Computing Machinery.
Information & Authors
Information
Published In
Journal of Engineering Mechanics
Volume 149 • Issue 3 • March 2023
Copyright
© 2023 American Society of Civil Engineers.
History
Received: Aug 14, 2022
Accepted: Nov 21, 2022
Published online: Jan 12, 2023
Published in print: Mar 1, 2023
Discussion open until: Jun 12, 2023
Authors
Metrics & Citations
Metrics
Citations
Download citation
If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download.