An Over-parameterized Exponential Regression
share
Over the past few years, there has been a significant amount of research focused on studying the ReLU activation function, with the aim of achieving neural network convergence through over-parametrization. However, recent developments in the field of Large Language Models (LLMs) have sparked interest in the use of exponential activation functions, specifically in the attention mechanism. Mathematically, we define the neural function F: ℝ^d × m×ℝ^d →ℝ using an exponential activation function. Given a set of data points with labels {(x_1, y_1), (x_2, y_2), …, (x_n, y_n)}⊂ℝ^d ×ℝ where n denotes the number of the data. Here F(W(t),x) can be expressed as F(W(t),x) := ∑_r=1^m a_r exp(⟨ w_r, x ⟩), where m represents the number of neurons, and w_r(t) are weights at time t. It's standard in literature that a_r are the fixed weights and it's never changed during the training. We initialize the weights W(0) ∈ℝ^d × m with random Gaussian distributions, such that w_r(0) ∼𝒩(0, I_d) and initialize a_r from random sign distribution for each r ∈ [m]. Using the gradient descent algorithm, we can find a weight W(T) such that F(W(T), X) - y _2 ≤ϵ holds with probability 1-δ, where ϵ∈ (0,0.1) and m = Ω(n^2+o(1)log(n/δ)). To optimize the over-parameterization bound m, we employ several tight analysis techniques from previous studies [Song and Yang arXiv 2019, Munteanu, Omlor, Song and Woodruff ICML 2022].
READ FULL TEXT