Editor's Note: This article originally appeared in SIAM News on June 01, 2022 (https://sinews.siam.org/Details-Page/dynamics-based-machine-learning-for-nonlinearizable-phenomena).
Machine learning (ML) has been an inspiring development for all areas of applied science, with numerous success stories in static learning environments like image, pattern, and speech recognition. Yet effective modeling of dynamical phenomena—such as nonlinear vibrations of solids and transitions in fluids—remains a challenge for ML, which tends to produce overly complex and uninterpretable dynamic models that are not reliable outside of their training range. A recent approach, however, integrates advanced dynamical systems concepts into elementary ML, ultimately yielding fast and accurate reduced-order models for nonlinear dynamics.
The idea—which we call dynamics-based machine learning (DBML)—is to learn models directly from phase space structures that are inferred from data. Systems with very different physics often display the same key invariant sets in their phase spaces; instead of fitting models to individual trajectories (which are sensitive to perturbations and parameter changes anyway), robust reduced-order modeling should therefore target structurally stable invariant sets. DBML focuses specifically on identifying the dynamics of ubiquitous, low-dimensional attracting invariant manifolds, which were first noted in the nonlinear vibrations literature [6]. Subsequent work in dynamical systems theory independently established the existence and properties of these manifolds, even for infinite-dimensional systems [1]. The forthcoming formulation, a higher-dimensional computational algorithm, and a data-driven implementation of these results have only appeared very recently [2-4].
Figure 1. Schematics of (1a) linear versus (1b) nonlinear model reduction near an attracting fixed point \(\mathcal{M}_{0}\). Figure adapted from [2].
DBML assumes the existence of at least one stationary state \(\mathcal{M}_0\) for a dynamical system, which we take here to be finite dimensional for simplicity. To further simplify the situation, we only consider the case wherein \(\mathcal{M}_0\) is an attracting fixed point; similar results hold for repelling fixed points, periodic orbits, and quasiperiodic steady states. The linearized dynamical system at \(\mathcal{M}_0\) will admit eigenspaces \(E_{j}\) that are spanned by generalized eigenvectors of its \(j\)th distinct eigenvalue \(\lambda_{j}\). We can order these eigenspaces by their increasing real parts, so that \(\mathrm{Re}\,\lambda_{j}<\mathrm{Re}\,\lambda_{j+1}\). As a consequence, solutions of the linearized system within \(E_{j}\) decay to the fixed point \(\mathcal{M}_0\) increasingly quickly as the index \(j\) grows.
By grouping some of the \(E_{j}\) eigenspaces together if necessary, we can build a hierarchy \(E^{1}\subset E^{2}\subset\ldots\) of spectral subspaces (see Figure 1a). We construct these spectral subspaces from eigenspaces in a manner that ensures that all \(E^{j}\) are non-resonant. In particular, no positive-integer linear combination of the eigenvalues in \(E^{j}\) should equal any eigenvalue that falls outside of \(E^{j}\). Each \(E^{j}\) thus contains linearized solutions that do not exchange energy via resonances with higher members of the spectral subspace hierarchy.
The subspace \(E^{j}\) serves as an observed attractor for typical linearized trajectories until the components of those trajectories in \(E^{j}-E^{j-1}\) die out. At that point, \(E^{j-1}\) becomes the observed attractor (see Figure 1a). The reduced dynamics on \(E^{j}\) therefore provide the best possible reduced model of the linearized dynamics if we wish to filter out transients that are associated with all stronger decay exponents \(\mathrm{Re}\,\lambda_{\ell}\) for \(\ell>j\).
Figure 2. Equation- and data-based predictions for the forced response curve (FRC) of a finite element model of an aircraft wing. Each trajectory integration in the forced finite element model (black cross) requires approximately four days to cover roughly 20 seconds of physical model time. In contrast, an equation-driven, spectral submanifold (SSM)-based prediction by SSMTool [4] (red curve) for the full FRC takes about 40 minutes. Finally, a purely data-driven FRC prediction by SSMLearn [2] (blue curve) that is trained on a single unforced trajectory takes about five minutes. Dashed portions of the SSM-based FRC predictions, which indicate unstable periodic response, are unavailable to direct numerical integration. Figure courtesy of the authors.
The fundamental result of spectral submanifold (SSM) theory is that this hierarchy of observed linear attractors also persists in a smoothly deformed form within the full nonlinear dynamical system. Specifically, a nested family of SSMs \(W(E^{1})\subset W(E^{2})\subset\ldots\) exists such that \(W(E^{j})\) is invariant under the full dynamics, has the same dimension as \(E^{j}\), and is tangent to \(E^{j}\) at the steady state \(\mathcal{M}_{0}\). These SSMs are not unique; they share their invariance, dimensionality, and tangency to \(E^{j}\) with infinitely many other manifolds. Under the addition of small periodic or quasiperiodic forcing, both \(\mathcal{M}_{0}\) and its SSMs persist smoothly and inherit the time dependence of the forcing.
Therefore, SSM-reduced dynamics provide a hierarchy of mathematically exact low-dimensional models for nonlinearizable behavior — even with the addition of moderate external forcing. Such behavior includes coexisting steady states, transitions among them, and chaotic dynamics. SSM-reduced models can be computed in seconds or minutes and reveal the details of nonlinearizable, damped-forced responses in mechanical systems with tens or even hundreds of thousands of degrees of freedom. For example, the red curve in Figure 2 traces an accurate and highly accelerated prediction of forced response from a two-dimensional reduced model on \(W(E^{1})\) for a 267,840-dimensional finite element model of an aircraft wing. Such a numerical prediction is currently impossible for even the most advanced numerical continuation packages [4].
The SSMTool computations in Figure 2 require explicit knowledge of nonlinearities in the governing equations, which is not available from commercial finite element codes. Even the evaluation of functions that implicitly define the nonlinearities is costly. The prohibitive expense for long-term simulations of individual trajectories means that model reduction is unavoidable.
Figure 3. Predictions for the forced horizontal oscillations of a sloshing fluid’s center of mass by a model that is trained on unforced experimental data. Figure adapted from [2] and left image courtesy of Kerstin Avilla.
One possible workaround is a fully data-driven algorithm for SSM construction [2]. This algorithm—which is implemented in an open-source MATLAB package called SSMLearn—uses data to identify the dimension and spectrum of the dominant spectral subspace \(E^{j}\). The procedure then utilizes regression to reconstruct the SSM in the observable space and computes a sparse normal form for the SSM-reduced dynamics.
This approach yields previously unthinkable computational speed-ups for dynamic finite element simulations. One can simply learn the unforced normal form on SSMs from a small number of decaying, unforced trajectories, then use these low-dimensional models to predict full bifurcation curves of the forced response without any simulation. The blue curve in Figure 2 is an example of this type of nonintrusive, data-driven model reduction, which yields remarkably close agreement with the exact analytic predictions from SSMTool. Here, SSMLearn was trained on a single unforced trajectory and predicted the full forced response curve in only five minutes.
Figure 4. Data-driven spectral submanifold (SSM)-based model and predictions for global transitions between stable and unstable states in a Couette flow. Figure adapted from [5].
SSM-based model reduction has multiple other uses as well. It is equally applicable to experimental data with arbitrary physics, such as sloshing dynamics in surface-wave experiments that are relevant in the design of tanks on cargo ships and commercial trucks (see Figure 3). Data-driven SSM reduction also provides low-dimensional reduced models for global transitions in infinite-dimensional nonlinear evolution equations. Such transitions occur, for example, in the Navier-Stokes equations for planar Couette flows, which admit multiple steady states beyond their stable, constant-shear base state (see Figure 4).
Here we have illustrated a physically diverse group of dynamical data sets from which DBML constructs accurate and predictive reduced-order models for nonlinearizable dynamics on SSMs. These dynamics display coexisting stable and unstable steady states with transitions among them, which cannot be simultaneously captured by a linear model. Promising ongoing extensions of SSM theory to more general \(\mathcal{M}_{0}\) sets, external forcing types, and non-smooth effects will further enhance the power of DBML.
References
[1] Cabré, X., Fontich, E., & de la Llave, R. (2003). The parameterization method for invariant manifolds I: Manifolds associated to non-resonant subspaces. Indiana Univ. Math. J., 52(2), 283-328.
[2] Cenedese, M., Axås, J., Bäuerlein, B., Avila, K., & Haller, G. (2022). Data-driven modeling and prediction of non-linearizable dynamics via spectral submanifolds. Nat. Commun., 13, 872.
[3] Haller, G., & Ponsioen, S. (2016). Nonlinear normal modes and spectral submanifolds: Existence, uniqueness and use in model reduction. Nonlin. Dynam., 86, 1493-1534.
[4] Jain, S., & Haller, G. (2021). How to compute invariant manifolds and their reduced dynamics in high-dimensional finite-element models. Nonlin. Dynam., 107, 1417-1450.
[5] Kaszás, B., Cenedese, M., & Haller, G. (2022). Dynamics-based machine learning of transitions in Couette flow. Preprint, arXiv:2203.13098.
[6] Shaw, S.W., & Pierre, C. (1993). Normal modes for non-linear vibratory systems. J. Sound Vibr., 164(1), 85-124.
George Haller is a professor of mechanical engineering at ETH Zürich, where he holds the Chair in Nonlinear Dynamics. His group works on various aspects of nonlinear dynamical systems that are defined by data sets. Shobhit Jain and Mattia Cenedese are postdoctoral researchers in this group, where they study nonlinear model reduction for equations and data.