We prove that the gradient descent map is non-singular for any neural network using piecewise analytic activation functions.