These notes represent my own study jottings and learning synthesis, not an authoritative or foolproof standard. If you find them helpful, feel free to use them. If you spot anything that could be clearer or have suggestions for additional concepts to include, issues and pull requests are warmly welcome.
Neural Networks & Deep Learning Foundations
Computational graphs, tensors, and the autodiff training loop
06. Neural Networks & Deep Learning Foundations
Computational graphs, the forward-backward loop, and the mechanics of automatic differentiation.
A neural network is not a mysterious new kind of learning. It is a composition of simple functions, fitted by the same gradient-based optimization used elsewhere in this handbook. What makes it different is scale: many layers, many parameters, and a graph structure that would be impossible to differentiate by hand.
This chapter builds the machinery one piece at a time. The training loop shows how the pieces fit together. Forward propagation evaluates the network. The chain rule explains how sensitivity travels backward. Backpropagation turns that into an algorithm. Gradients, computational graphs, automatic differentiation, and tensors explain the objects and operations involved. Gradient accumulation is the practical technique that decouples batch size from memory.
Read the training loop first, then follow the dependency order. The later sections assume the earlier ones.