Thread · HN Continuity

This is probably the most important concept for "normies" to understand about AI, IMO. It's the stochastic brother of the deterministic Church-Turing thesis. Any function that can be computed can be computed on any computer. And that function can be approximated to an arbitrary degree of precision with a DNN.

The real kicker is DNNs are much easier to program than CPUs because they don't require a closed-form description ("a program") of the function to be approximated; you just throw a bunch of input/output pairs at the model, compute loss, backprop and update weights, repeat.

Hence the unslakeable thirst for input/output pairs, i.e. data.

> In the field of machine learning, the universal approximation theorems (UATs) state that

> neural networks with a certain structure can, in principle, approximate any continuous

> function to any desired degree of accuracy. These theorems provide a mathematical

> justification for using neural networks, assuring researchers that a sufficiently large or

> deep network can model the complex, non-linear relationships often found in real-world data.[1][2]

> The best-known version of the theorem applies to feedforward networks with a single hidden

> layer. It states that if the layer's activation function is non-polynomial (which is true

> for common choices like the sigmoid function or ReLU), then the network can act as a

> "universal approximator." Universality is achieved by increasing the number of neurons in

> the hidden layer, making the network "wider." Other versions of the theorem show that

> universality can also be achieved by keeping the network's width fixed but increasing its

> number of layers, making it "deeper."

<a href="https://en.wikipedia.org/wiki/Universal_approximation_theorem" rel="nofollow">https://en.wikipedia.org/wiki/Universal_approximation_theore...