Deep Network Approximation Characterized by Number of Neurons
share
This paper quantitatively characterizes the approximation power of deep feed-forward neural networks (FNNs) in terms of the number of neurons, i.e., the product of the network width and depth. It is shown by construction that ReLU FNNs with width and depth 9L+12 can approximate an arbitrary Hölder continuous function of order α with a Lipschitz constant ν on [0,1]^d with a tight approximation rate 5(8√(d))^αν N^-2α/dL^-2α/d for any given N,L∈^+. The constructive approximation is a corollary of a more general result for an arbitrary continuous function f in terms of its modulus of continuity ω_f(·). In particular, the approximation rate of ReLU FNNs with width and depth 9L+12 for a general continuous function f is 5ω_f(8√(d) N^-2/dL^-2/d). We also extend our analysis to the case when the domain of f is irregular or localized in an ϵ-neighborhood of a d_M-dimensional smooth manifold M⊆ [0,1]^d with d_M≪ d. Especially, in the case of an essentially low-dimensional domain, we show an approximation rate 3ω_f(4ϵ1-δ√(dd_δ))+5ω_f(16d(1-δ)√(d_δ)N^-2/d_δL^-2/d_δ) for ReLU FNNs to approximate f in the ϵ-neighborhood, where d_δ=(d_M (d/δ)δ^2) for any given δ∈(0,1). Our analysis provides a general guide for selecting the width and the depth of ReLU FNNs to approximate continuous functions especially in parallel computing.
READ FULL TEXT