Nonlinear Regression without i.i.d. Assumption

1 Introduction

In statistics, linear regression is a linear approach for modelling the relationship between an explaining variable

y

and one or more explanatory variables denoted

x

y = w^{T} x + b .

The parameters $w, b$

can be estimated via the method of least squares.

The first clear and concise exposition of the method of least squares was published by Legendre[6] in 1805. Later in 1809, Gauss published his method of calculating the orbits of celestial bodies. In that work he claimed to have been in possession of the method of least squares since 1795. Here is the basic theorem of linear regression.

Theorem 1.1.

Suppose ${(x_{i}, y_{i})}_{i = 1}^{m}$ are drawn from the linear model with

y_{i} = w^{T} x_{i} + b + ε_{i},

where the error terms $ε_{1}, ε_{2}, \dots, ε_{m}$ are independent Gaussian variables with mean $0$

and variance

σ^{2}

. Denote

A = ⎛ ⎜ ⎜ ⎜ ⎝ \begin{matrix} x_{11} & x_{12} & \dots & x_{1 d} & 1 x_{21} & x_{22} & \dots & x_{2 d} & 1 \dots & \dots & \dots & \dots & \dots x_{m 1} & x_{m 2} & \dots & x_{m d} & 1 \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎠, c = ⎛ ⎜ ⎜ ⎜ ⎝ \begin{matrix} y_{1} y_{2} \dots y_{m} \end{matrix} ⎞ ⎟ ⎟ ⎟ ⎠ .

Then the result of least square is

(w_{1}, w_{2}, \dots, w_{d}, b)^{⊤} = A^{+} c .

Here, $A^{+}$ is the Moore-Penrose inverse⁴⁴4For the definition and property of Moore-Penrose inverse, see [1]. of $A$ .

In the above theorem, the errors $ε_{1}, ε_{2}, \dots, ε_{m}$ are assumed to be independent Gaussian variables. Therefore, $y_{1}, y_{2}, \dots, y_{m}$ are also independent Gaussian variables.

When the i.i.d. (independent and identically distributed) assumption is not satisfied, the usual method of least squares does not work well. This can be illustrated by the following example.

Example 1.1.

Suppose the sample data (training set) is

(x_{1}, y_{1}) = (x_{2}, y_{2}) = \dots = (x_{500}, y_{500}) = (0.15, 1.48),

(x_{501}, y_{501}) = (x_{502}, y_{502}) = \dots = (x_{1000}, y_{1000}) = (0.43, 1.45),

(x_{1001}, y_{1001}) = (x_{1002}, y_{1002}) = \dots = (x_{1500}, y_{1500}) = (0.04, 1.59),

(x_{1501}, y_{1501}) = (1.23, 3.01), (x_{1502}, y_{1502}) = (0.63, 2.89), (x_{1503}, y_{1503}) = (1.64, 4.54),

(x_{1504}, y_{1504}) = (0.98, 3.32), (x_{1505}, y_{1505}) = (1.92, 5.0), (x_{1506}, y_{1506}) = (1.26, 3.96),

(x_{1507}, y_{1507}) = (1.77, 3.92), (x_{1508}, y_{1508}) = (1.1, 2.8), (x_{1509}, y_{1509}) = (1.22, 2.84),

(x_{1510}, y_{1510}) = (1.48, 4.52), (x_{1511}, y_{1511}) = (0.71, 3.17), (x_{1512}, y_{1512}) = (0.77, 2.59),

(x_{1513}, y_{1513}) = (1.89, 5.1), (x_{1514}, y_{1514}) = (1.31, 3.17), (x_{1515}, y_{1515}) = (1.31, 2.91),

(x_{1516}, y_{1516}) = (1.63, 4.02), (x_{1517}, y_{1517}) = (0.56, 1.79) .

Following is the result of the usual least square.

y = 0.4711 * x + 1.4258.

We can see from the graph that most of the sample data deviates from the regression line. The main reason is that $(x_{1}, y_{1}), (x_{2}, y_{2}), \dots, (x_{500}, y_{500})$ are the same sample and the i.i.d. condition is violated.

In light of this, Lin[7] study the linear regression without i.i.d. condition by using the nonlinear expectation framework laid down by Peng[9]

. They split the training set into several groups and in each group the i.i.d. condition can be satisfied. The average loss is used for each group and the maximum of average loss among groups is used as the final loss function. They show that the linear regression problem under the nonlinear expectation framework is reduced to the following mini-max problem.

min w, b max 1 \leq j \leq N \frac{1}{M} M \sum l = 1 (w^{T} x_{j l} + b - y_{j l})^{2} .

(1)

They propose a genetic algorithm to solve this problem. However, the algorithm does not work well generally.

Motivated by Lin[7] and Peng[9]’s work, we consider nonlinear regression problems without the assumption of i.i.d. in this paper. We propose a correspondent mini-max problems and outline a numerical algorithm for solving this problem based on the work of Kiwiel[4]. Meanwhile, problem (1) in Lin’s paper can also be well solved by such an algorithm. We also have done some experiments in regression and machine learning problems.

2 Nonlinear Regression without i.i.d. Assumption

Nonlinear regression is a form of regression analysis in which observational data are modeled by a function which is a nonlinear combination of the model parameters and depends on one or more explanatory variables. (see e.g.

[10])

Suppose the sample data (training set) is

S = {(x_{1}, y_{1}), (x_{2}, y_{2}), \dots, (x_{m}, y_{m})},

where $x_{i} \in X$ and $y_{i} \in Y$ . $X$ is called the input space and $Y$ is called the output (label) space. The goal of nonlinear regression is to find (learn) a function $g^{θ} : X \to Y$ from the hypothesis space ${g^{λ} : X \to Y | λ \in Λ}$ such that $g^{θ} (x_{i})$ is as closer to $y_{i}$ as possible.

The closeness is usually characterized by a loss function $φ$ such that $φ \geq 0$ and

φ (g^{θ} (x_{1}), y_{1}, \dots, g^{θ} (x_{m}), y_{m}) \to 0

if and only if

g^{θ} (x_{i}) - y_{i} \to 0, 1 \leq i \leq m .

Then the learning problem is reduced to an optimization problem of minimizing $φ$ .

Following are two kinds of loss functions, namely, the average loss and the maximal loss.

φ_{2} = \frac{1}{m} m \sum j = 1 (g^{θ} (x_{j}) - y_{j})^{2} .

φ_{\infty} = max 1 \leq j \leq m (g^{θ} (x_{j}) - y_{j})^{2} .

The average loss is popular, particularly in machine learning, since it can be conveniently minimized using online algorithms, which process few instances at each iteration. The idea behinds the average loss is to learn a function that performs equally well for each training point. However, when the i.i.d. assumption is not satisfied, the average loss way may become a problem.

To overcome this difficulty, we use the max-mean as the loss function. First, we split the training set into several groups and in each group the i.i.d. condition can be satisfied. Then the average loss is used for each group and the maximum of average loss among groups is used as the final loss function. We propose the following mini-max problem for nonlinear regression problem.

min θ max 1 \leq j \leq N \frac{1}{n_{j}} n_{j} \sum l = 1 (g^{θ} (x_{j l}) - y_{j l})^{2} .

(2)

Here, $n_{j}$ is the number of samples in group $j$ .

Problem (2) is a generalization of problem (1). Next, we will give a numerical algorithm which solves problem (2).

Remark 2.1.

Peng and Jin[2] put forward a max-mean method to give the parameter estimation when the usual i.i.d. condition is not satisfied. They show that if $Z_{1}, Z_{2}, \dots, Z_{k}$ are drawn from the maximal distribution $M_{[μ - -, ¯ ¯ ¯ μ]}$

and are nonlinearly independent, then the optimal unbiased estimation for

¯ ¯ ¯ μ

max {Z_{1}, Z_{2}, \dots, Z_{k}} .

This fact, combined with the Law of Large Numbers (Theorem 19 in

[2]) leads to the max-mean estimation of

μ

. We borrow this idea and use the max-mean as the loss function for the nonlinear regression problem.

3 Algorithm

Problem (2

) is a mini-max problem. The mini-max problems arise in different kinds of mathematical fields, such as game theory and the worst-case optimization. The general mini-max problem is described as

min u \in R^{N} max v \in V h (u, v) .

(3)

Here, $h$ is continuous on $R^{n} \times V$ and differentiable with respect to $u$ . $V$ is a compact subset of $R^{d}$ .

Problem (3) was considered theoretically by Klessig and Polak[5] in 1973 and Panin[8] in 1981. Later in 1987, Kiwiel[4] gave a concrete algorithm for problem (3).

Kiwiel linearized $h$ at each iterative point $u_{k}$ and obtain the convex approximation for $u \mapsto max v \in V h (u, v)$ as

^h(u)=maxv∈V{h(uk,v)+⟨∇h(uk,v),u−uk⟩}.

The next step is to find $u_{k + 1}$ , which minimizes $^h (u)$ .

In general, $^h$ is not strictly convex with respect to $u$ , thus may not admit a minimum. To overcome this difficulty, he added a regularization term and the minimization problem is reduced to

minu∈RN{^h(u)+12|u−uk|2}.

It can be converted to the following form

min d \in R^{N} {max v \in V {h (u_{k}, v) + ⟨ \nabla h (u_{k}, v), d ⟩} + \frac{1}{2} | d |^{2}},

which is equivalent to

minimize (\frac{1}{2} | d |^{2} + a)

over all $(d, a) \in R^{N} \times R$ satisfying

h (u_{k}, v) + ⟨ \nabla h (u_{k}, v), d ⟩ \leq a, \forall v \in V .

By duality theory, the above problem is further transformed into

min ⎛ ⎝ \frac{1}{2} {∣ ∣ ∣ ∣ \sum v \in V λ_{v} \nabla_{u} h (u_{k}, v) ∣ ∣ ∣ ∣}_{v}^{2} - \sum v \in V λ_{v} h (u_{k}, v) ⎞ ⎠

over all $λ = (λ_{v})$ with finitely many $λ_{v} \neq 0$ and

λ_{v} \geq 0, \forall v \in V,

\sum v \in V λ_{v} = 1.

Our problem (2) is a special case of problem (3) with $V = {1, 2, \dots, N}$ and

h (u, j) = \frac{1}{n_{j}} n_{j} \sum l = 1 (g^{u} (x_{j l}) - y_{j l})^{2} .

Denote $f_{j} (u) = h (u, j)$ and we give a numerical algorithm for the following mini-max problem.

min u \in R^{n} max 1 \leq j \leq N f_{j} (u) .

(4)

Suppose each $f_{j}$ is differentiable with respect to $u$ and denote

Φ (u) = max 1 \leq j \leq N f_{j} (u) .

Step 1. Initialization

Select arbitrary $u_{0} \in R^{n}$ . Set $k = 0$ and

termination accuracy ξ = 10^{- 6},

linear approximation parameter m = 2 \times 10^{- 4},

line search parameter c = 10^{- 4},

stepsize factor σ = 0.5.

Step 2. Direction Finding

Assume that we have chosen $u_{k}$ . For $(u_{k}, Φ (u_{k}))$ ,

Step 2.1. Initialization

Set $i = 1, v_{1} = 1$ . Compute

p_{0} = \nabla f_{v_{1}} (u_{k}), θ_{0} = f_{v_{1}} (u_{k}) .

Step 2.2. Weight Finding

Solve the following quadratic optimization problem.

μ_{i} = arg min μ \in R {\frac{1}{2} ∥ (1 - μ) p_{i - 1} + μ \nabla f_{v_{i}} (u_{k}) ∥^{2} - (1 - μ) θ_{i - 1} - μ f_{v_{i}} (u_{k})} .

Set

p_{i} = (1 - μ_{i}) p_{i - 1} + μ_{i} \nabla f_{v_{i}} (u_{k}), θ_{i} = (1 - μ_{i}) θ_{i - 1} + μ_{i} f_{v_{i}} (u_{k}) .

Ψ_{i} = - (∥ p_{i} ∥^{2} + Φ (u_{k}) - θ_{i}) .

If $Ψ_{i} \geq - ξ$ , stop; otherwise, goto Step 2.3.

Step 2.3. Primal Optimality Testing

Set $d_{i} = - p_{i}$ ,

v_{i + 1} = arg max 1 \leq j \leq N {f_{j} (u_{k}) + ⟨ \nabla f_{j} (u_{k}), d_{i} ⟩} .

f_{v_{i + 1}} (u_{k}) + ⟨ \nabla f_{v_{i + 1}} (u_{k}), d_{i} ⟩ - Φ (u_{k}) \leq m Ψ_{i},

goto Step 2.4; otherwise, set $i = i + 1$ , and goto Step 2.2.

Step 2.4. Return

Set

d_{k} = - p_{i}, η_{k} = Ψ_{i} .

If $Ψ_{i} \geq - ξ$ , stop; otherwise, goto Step 3.

Step 3. Line Search

Set

I_{k} = {σ^{j} : Φ (u_{k} + σ^{j} d_{k}) - Φ (u_{k}) \leq c σ^{j} η_{k}, j = 0, 1, \dots}

and

α_{k} = max I_{k} .

Set

u_{k + 1} = u_{k} + α_{k} d_{k}, k = k + 1

and goto Step 2.

4 Experiment

4.1 The Linear Regression Case

Example 1.1 can be numerically well solved by the above algorithm with

f_{j} (w, b) = (w x_{j} + b - y_{j})^{2}, j = 1, 2, \dots, 1517.

The corresponding optimization problem is

min w, b max 1 \leq j \leq 1517 (w x_{j} + b - y_{j})^{2} .

The numerical result using the algorithm in section 3 is

y = 1.7589 * x + 1.2591.

The following picture summarize the result. It can be seen that the method using maximal loss function (black line) performs better than the traditional least square method (pink line).

Figure 2: New Method v.s. Traditional Method

4.2 Machine Learning

In this case, we use the MNIST database

⁵⁵5http://yann.lecun.com/exdb/mnist/

of handwritten digits to perform the experiment. Many machine learning models cater to the identification of handwritten digits. We use the multi-layer perception model. Three hidden layers along with an input layer and an output layer are used. The number of neurons in each layer is 784,50,20,12,10, respectively.

The training data is chosen as follows. Firstly, we choose 1000 training data and split them randomly into $10$ groups, named $G_{1}, G_{2}, \dots, G_{10}$ . Each group has 100 i.i.d. samples. Then we set

A_{1} = A_{2} = \dots = A_{91} = G_{1}, A_{92} = G_{2}, \dots, A_{100} = G_{10} .

Two different method are applied. One is using the average group loss

\frac{1}{100} 100 \sum j = 1 {\frac{1}{100} 100 \sum l = 1 | h^{θ} (x_{j l}) - y_{j l} |^{2}} .

The other is using the maximal group loss

max 1 \leq j \leq 100 {\frac{1}{100} 100 \sum l = 1 | h^{θ} (x_{j l}) - y_{j l} |^{2}} .

10000 additional test data are used to test these two models. The accuracy of the method using maximal loss is $65.3 %$ , while the accuracy of the method using average loss is $58.1 %$ .

In this experiment, the whole training set is not i.i.d.. On the other hand, each subgroup is i.i.d.. It turns out that the method using maximal loss performs better than the method using average loss. In the last 20 years, deep learning using many more hidden layers and other machine learning algorithm has achieved accuracy over

90 %

in the handwriting recognition and other problems. However, we think that the method in this paper can also improve the performance when the training set is not i.i.d..

5 Conclusion

In this paper, we consider a class of nonlinear regression problems without the assumption of being independent and identically distributed. We propose a correspondent mini-max problem for nonlinear regression and outline a numerical algorithm. Such an algorithm can be applied in regression and machine learning problems, and yield better results than traditional regression and machine learning methods.

Acknowledgement

The authors would thank Professor Shige Peng for useful discussions.

References

[1] Ben-Israel, Adi, Greville, Thomas N.E. (2003). Generalized inverses: Theory and applications (2nd ed.). New York, NY: Springer.
[2] Hanqing Jin, Shige Peng (2016). Optimal Unbiased Estimation for Maximal Distribution. https://arxiv.org/abs/1611.07994.
[3] Kendall, M. G., Stuart, A. (1968). The Advanced Theory of Statistics, Volume 3: Design and Analysis, and Time-Series (2nd ed.). London: Griffin.
[4] Kiwiel, K.C. (1987). A Direct Method of Linearization for Continuous Minimax Problems. Journal of Optimization Theory and Applications, 55, 271-287.
[5] Klessig, R. and E. Polak (1973). An Adaptive Precision Gradient Method for Optimal Control. SIAM Journal on Control, 11, 80-93.
[6] Legendre, Adrien-Marie (1805). Nouvelles methodes pour la determination des orbites des cometes.
[7] Lin L., Shi Y., Wang X., and Yang S (2013). Sublinear Expectation Linear Regression. Statistics.
[8] Panin, V.M. (1981). Linearization Method for Continuous Min-max Problems. Kibernetika, 2, 75-78.
[9]
Peng, S. (2005). Nonlinear expectations and nonlinear Markov chains.
Chin. Ann. Math., 26B(2), 159-184.
[10] Seber, G. A. F., Wild, C. J. (1989). Nonlinear Regression. New York: John Wiley and Sons.

Nonlinear Regression without i.i.d. Assumption

Authors

Analytic Continued Fractions for Regression: A Memetic Algorithm Approach

Extensions of Morse-Smale Regression with Application to Actuarial Science

Traditional Machine Learning for Pitch Detection

Estimating nonlinear regression errors without doing regression

A Comparative Study of Pairwise Learning Methods based on Kernel Ridge Regression

Interpretable Reinforcement Learning with Ensemble Methods

Horseshoe Regularization for Machine Learning in Complex and Deep Models

1 Introduction

Theorem 1.1.

Example 1.1.

2 Nonlinear Regression without i.i.d. Assumption

Remark 2.1.

3 Algorithm

4 Experiment

4.1 The Linear Regression Case

4.2 Machine Learning

5 Conclusion

Acknowledgement

References

Nonlinear Regression without i.i.d. Assumption

Authors

Analytic Continued Fractions for Regression: A Memetic Algorithm Approach

Extensions of Morse-Smale Regression with Application to Actuarial Science

Traditional Machine Learning for Pitch Detection

Estimating nonlinear regression errors without doing regression

A Comparative Study of Pairwise Learning Methods based on Kernel Ridge Regression

Interpretable Reinforcement Learning with Ensemble Methods

Horseshoe Regularization for Machine Learning in Complex and Deep Models

This week in AI

1 Introduction

Theorem 1.1.

Example 1.1.

2 Nonlinear Regression without i.i.d. Assumption

Remark 2.1.

3 Algorithm

4 Experiment

4.1 The Linear Regression Case

4.2 Machine Learning

5 Conclusion

Acknowledgement

References