People make far too big a deal of the universal function approximation property of a hidden-layer neural network. Universality should be a basic property of any decent interpolation method.
Piecewise linear regression is a universal function approximator.
And GMMs and 4th order DAEs and a bunch of other things. The really interesting thing is if the sensitivity of the weights to the loss function is a simple function. Best case would be a convex optimization problem, e.g. like SVMs
Piecewise linear regression is a universal function approximator.