Search papers, labs, and topics across Lattice.
This paper establishes that a single ReLU recurrent neural network (RNN) with fixed weights and hidden dimensions can uniformly approximate any continuous function on the interval [-1,1] by simply allowing it to run longer. The authors introduce the Turing machine with neural units (TMNU) as an intermediate model that balances algorithmic flexibility with the constraints necessary for RNN simulation, leading to convergence rates that align with polynomial approximation schemes. Additionally, they provide minimax lower bounds to demonstrate that the required runtime is a fundamental aspect of this fixed-network approximation approach, rather than an artifact of the proof.
A single fixed RNN can achieve any desired accuracy for continuous functions by simply running longer, challenging the need for new networks with improved target accuracy.
Classical approximation theorems ask for a new neural network whenever the target accuracy is improved. This paper studies the opposite possibility: can the network be chosen once and for all, and can accuracy be bought only by letting it run longer? We prove that this is possible for every continuous function on [-1,1]. More precisely, each such function is uniformly approximated by the time evolution of a single ReLU recurrent neural network with fixed weights and fixed hidden dimension. The mechanism behind the construction is a new intermediate model, the Turing machine with neural units (TMNU). This model retains the algorithmic freedom needed to implement polynomial approximation schemes, while remaining rigid enough to be simulated by RNNs with explicit bounds on hidden dimension and weight magnitude. The resulting convergence rates reflect the underlying polynomial approximation rates. We complement the construction with minimax lower bounds showing that runtime is not merely a proof artifact, but an unavoidable resource in this fixed-network approximation paradigm.
