DeepAI
Log In Sign Up

Dimension Reduction Using Active Manifolds

02/07/2018
by   Robert A. Bridges, et al.
0

Scientists and engineers rely on accurate mathematical models to quantify the objects of their studies, which are often high-dimensional. Unfortunately, high-dimensional models are inherently difficult, i.e. when observations are sparse or expensive to determine. One way to address this problem is to approximate the original model with fewer input dimensions. Our project goal was to recover a function f that takes n inputs and returns one output, where n is potentially large. For any given n-tuple, we assume that we can observe a sample of the gradient and output of the function but it is computationally expensive to do so. This project was inspired by an approach known as Active Subspaces, which works by linearly projecting to a linear subspace where the function changes most on average. Our research gives mathematical developments informing a novel algorithm for this problem. Our approach, Active Manifolds, increases accuracy by seeking nonlinear analogues that approximate the function. The benefits of our approach are eliminated unprincipled parameter, choices, guaranteed accessible visualization, and improved estimation accuracy.

READ FULL TEXT VIEW PDF

page 1

page 2

page 3

page 4

02/27/2019

Deep active subspaces -- a scalable method for high-dimensional uncertainty propagation

A problem of considerable importance within the field of uncertainty qua...
04/30/2019

Active Manifolds: A non-linear analogue to Active Subspaces

We present an approach to analyze C^1(R^m) functions that addresses limi...
08/08/2020

A Fully Bayesian Gradient-Free Supervised Dimension Reduction Method using Gaussian Processes

Modern day engineering problems are ubiquitously characterized by sophis...
06/08/2021

Sequential active learning of low-dimensional model representations for reliability analysis

To date, the analysis of high-dimensional, computationally expensive eng...
11/18/2020

Improving kriging surrogates of high-dimensional design models by Partial Least Squares dimension reduction

Engineering computer codes are often computationally expensive. To light...

1 Introduction

Scientists and engineers rely on accurate mathematical models to quantify the objects of their studies, which are often high-dimensional. Unfortunately, high-dimensional models are inherently difficult, i.e. when observations are sparse or expensive to determine. One way to address this problem is to approximate the original model with fewer input dimensions. Our project goal was to recover a function that takes inputs and returns one output, where is potentially large. For any given -tuple, we assume that we can observe a sample of the gradient and output of the function but it is computationally expensive to do so. This project was inspired by an approach known as Active Subspaces, which works by linearly projecting to a linear subspace where the function changes most on average. Our research gives mathematical developments informing a novel algorithm for this problem. Our approach, Active Manifolds, increases accuracy by seeking nonlinear analogues that approximate the function. The benefits of our approach are eliminated unprincipled parameter, choices, guaranteed accessible visualization, and improved estimation accuracy.

2 Related Work

Dimension reduction, broadly defined, is the mapping of potentially high dimensional data to a lower dimensional space. Dimension reduction techniques can be categorized into two main categories, projective methods and manifold modeling

[1]

. Dimension reduction techniques are widely used across many domains to analyze high-dimensional models or high-dimensional data sets because they allow important low-dimensional features to be extracted and allow for data visualization. The most commonly known and used projective method is Principal Component Analysis (see

[2]). The method that inspired our work, Active Subspaces, can also be considered a projective method. The Nyström method (see [3]

) and related variations rely on eigenvalue problems and compromise the bulk of manifold modeling techniques. Our method, Active Manifolds, departs from the use of projective and spectral methods but is a manifold modeling method.

2.1 Active Subspaces

We chose to study Active Subspaces because it is a dimension reduction technique that reduces the dimension of the input space while respecting the output, its applicability to a wide range of functions (), and because of its accessibility to scientists and engineers with a limited mathematical background. The Active Subspaces method finds lower-dimensional subspaces of the domain by finding the directions in which the function changes the most on average. The Active Subspaces method has two main limitations. First, many functions do not admit a linear active subspace, e.g. . Second, the linearity of active subspaces and projections is restrictive and can increase estimate error.

Below is a brief description of the Active Subspaces algorithm.

  1. Sample at random points

  2. Find the directions in which changes the most on average, the active subspace. This is done by computing the eigenvalue decomposition of the matrix

  3. Perform regression to estimate along the active subspace to obtain (this requires sampling at random points ).

  4. Given a new point , project to the active subspace and use to obtain the value .

3 Results

3.1 Theory

Recall that arc length of a curve is given by

Let , where . Assume is . We seek


over all functions , and (constant speed).

Notice the integrand can be expressed as

where is the angle between and . Trivially, this quantity is maximal when , indicating that and are collinear and point in the same direction. Thus,

(1)
Definition 3.1.

Let and and . We say that is an active manifold of if and only if, for all charts on , condition (1) is satisfied when .

Lemma 3.2.

Given and an initial value , there exists a unique solution to the system of first-order differential equations described in (1).

Proof.

Assume the region is compact and convex. Since is , satisfies the Lipschitz condition

for and some Lipschitz constant . These conditions are sufficient for the existence and uniqueness of a solution to (1) for a given initial value (see Chapter 6, Theorem 1 from [4]). ∎

For the following theorem, let

  • = range

  • be a fixed critical point of

  • be the deleted attracting basin of


[row sep=huge, column sep = huge] & X [rd, ”f”] [d, ”π”] &
R [ur, ”x(t)”] [r, ”π∘x”] & X/∼[r, ”~f”] & R Commutative diagram for Theorem 3.3

Theorem 3.3.
  1. [label=()]

  2. If is a solution to (1), then is a 1-dim. submanifold of .

  3. is a manifold.

  4. If and is a solution to (1) then imbeds into the manifold .

Proof.
  1. [label=()]

  2. Realize as the single chart for induced by , thus is a 1-dimensional submanifold of .

  3. Realize given by is continuous so is a manifold with a single chart .

  4. Realize is a bijection. It follows that is a diffeomorphism from to since charts on and are induced by .

Further, the Implicit Function Theorem implies that is a -dimensional manifold and orthogonally intersects . We refer to such a manifold as an Active Manifold (denoted ). Thus, we can realize an by a numerical solution to (1).

3.2 Active Manifold Algorithm Description

The Active Manifolds algorithm has three main procedures:

  1. Building the Active Manifold

  2. Approximating the function of interest,

  3. Projecting a point of interest to the Active Manifold

3.2.1 Building the Active Manifold

For a given function, where , we describe below a process to build a corresponding active manifold. The active manifold will be a one-dimensional curve in the hypercube that moves from a local minimum to a local maximum. We define a grid with spacing size then compute at each grid point. To build the active manifold, we use a modified gradient ascent/descent scheme with a nearest neighbor search.

  1. Construct an -dimensional grid with spacing size .

    1. [label*=0.]

    2. Compute at each grid point.

    3. Normalize samples.

  2. Given an initial starting point , use a gradient ascent/descent scheme with a nearest neighbor search to find a numerical solution to

    with the samples from step 1. The set is an active manifold on .

    1. [label*=0.]

    2. While the active manifold builds, save the number of steps and functional values corresponding to the closest grid point at each step as ordered lists and .

    3. Scale by so that

3.2.2 One-Dimensional Function Approximation

To obtain a one-dimensional approximation , perform regression on the points of . A major benefit of our method is that the set can be easily plotted and serves as a visual aid to help choose a best fit model.

3.2.3 Traversing the Level Set

Given a point , we would like to find , where and . This requires an iterative process that uses the orthogonal directions of to travel along the level set corresponding to , until we intersect the active manifold. For the following algorithm, we assume that has been normalized and tolerance and step size have been selected.

  1. Given a point , find for (closest point on the manifold to )

  2. Construct a vector from

    to , .

  3. Find .

  4. While , let .

  5. Parameterize the line segment, , between and , where is the next closest point on the manifold to .

  6. Determine such that is orthogonal to .

  7. Determine such that .

  8. Evaluate .

Figure 1: Schematic for Level Set Algorithm in when starting point is one step from manifold. Notation in schematic matches pseudo-algorithm above.

3.2.4 Algorithm Notes

  1. One may be enticed to minimize a distance function , for example

    by letting

    but this is computationally inefficient because now we must compute along with . Instead, we recommend computing and searching for the minimum or using some other nearest neighbor search.

  2. If , it is convenient to express , and is an orthonormal basis for , it is helpful to express

  3. It is convenient to let

  4. The point on for which is orthogonal to can be determined by solving for t in

    . Solving for gives

3.3 Empirical Results

For proof of concept and comparison to methods in [5], we proceed with data synthesized from two functions,

For each example, we are interested in how well two functions, one fit to the the Active Subspace (AS) and one fit to the Active Manifold (AM), recover the values of the function for points outside of the AS and AM. We build the AS and AM and calculate the average error for a set of random test points. The following experimental set up was observed for each example.

  1. Define a uniform grid on with 0.05 point spacing.

  2. Evaluate the gradient of each function, computed analytically, at each grid point.

  3. Build the AS and AM using the gradient.

  4. Fit the AS and AM with a polynomial ( quartic, quintic).

  5. Draw 100 random samples and map them to the AS and AM.

  6. Evaluate at each sample point and at the corresponding projection point.

Upon completing the experiments, average absolute errors between and were computed.

Manifold Error
Subspace Error

Notice that the AM reduces average absolute error, by an order of magnitude, in both examples.

In [6] the authors investigate an active subspace in a 5-dimensional single-diode solar cell model. We have followed by reproducing their results with their data, while also implementing the Active Manifold algorithm. Again, we are interested in comparing estimation errors. The experiment included 10,000 randomly sampled points from . This sample was partitioned into two randomly ordered sets containing 8,000 and 2,000 points, used for training and testing, respectively. After an eight fold Monte Carlo simulation, the mean of the average absolute error was computed.

Manifold Error
Subspace Error

Again, the AM reduces average absolute error, by an order of magnitude

4 Conclusions and Future Work

The primary issue to be addressed in future work is determining the most appropriate way to choose an active manifold. What happens when a level set never intersects the active manifold? Should a new manifold be chosen? For future work, we propose a parallel program implementation that may run the algorithm for two or more manifolds. Future work will also seek error bounds and computational complexity estimates for the algorithm.

Acknowledgements

  • This work was supported in part by the U.S. Department of Energy, Office of Science, Office of Workforce Development for Teachers and Scientists (WDTS) under the program SULI.

  • ORNL’s Cyber and Information Security Research Group (CISR)

References

  • [1] Christopher JC Burges. Dimension reduction: A guided tour. Now Publishers Inc, 2010.
  • [2] Karl Pearson. Liii. on lines and planes of closest fit to systems of points in space. The London, Edinburgh, and Dublin Philosophical Magazine and Journal of Science, 2(11):559–572, 1901.
  • [3] Sanjiv Kumar, Mehryar Mohri, and Ameet Talwalkar. Sampling methods for the nyström method.

    Journal of Machine Learning Research

    , 13(Apr):981–1006, 2012.
  • [4] G. Birkhoff and G.C. Rota. Ordinary differential equations. Blaisdell book in pure and applied mathematics. Blaisdell Pub. Co., 1969.
  • [5] Paul G Constantine. Active subspaces: Emerging ideas for dimension reduction in parameter studies, volume 2. SIAM, 2015.
  • [6] Paul G Constantine, Brian Zaharatos, and Mark Campanelli. Discovering an active subspace in a single-diode solar cell model.

    Statistical Analysis and Data Mining: The ASA Data Science Journal

    , 8(5-6):264–273, 2015.