Unit III Overview: Learning With Neural Networks

Artificial Neural Networks (ANNs) are inspired by the biological neural networks in animal brains. The modern deep learning revolution began with the 2012 ImageNet breakthrough by Krizhevsky, Sutskever, and Hinton (AlexNet), demonstrating that deep convolutional networks could dramatically outperform traditional methods.

Biological Inspiration

A biological neuron:

1. Receives signals from dendrites (inputs with varying weights)
2. Integrates signals in the cell body (weighted sum)
3. Fires an action potential if the sum exceeds a threshold (activation function)
4. Transmits signal via axon (output)

An artificial neuron computes: y = f(sum(w_i * x_i) + b) where f is the activation function.

Unit III Roadmap

Activation Functions

Universal Approximation Theorem

A feedforward network with a single hidden layer containing enough neurons can approximate any continuous function on a compact subset of R^n to arbitrary precision. This justifies why neural networks can, in theory, learn any pattern.

Exam-Ready Summary

• ANN: layers of weighted sums followed by non-linear activation functions
• Universal Approximation: 1 hidden layer suffices in theory; depth helps in practice
• ReLU is the default activation for hidden layers (avoids vanishing gradient)
• Depth provides compositional representations — exponentially more efficient than shallow networks
• Modern deep learning relies on GPUs, large datasets, and architectural innovations (BN, dropout, skip connections)