Backpropagation
11 study materials · 5 sections
This course provides a comprehensive exploration of Backpropagation, the fundamental algorithm for training neural networks. It covers the historical evolution of the method from its roots in sensitivity analysis and optimal control to its modern application in deep learning. Students will examine the mathematical foundations of gradient computation, its extension to recurrent and recursive structures, and advanced optimization techniques for large-scale data.
Course Sections
Historical Context and the Backwards Method
Key concepts: Interpretative Flexibility · Perceptron Controversy · Ordered Derivatives · Backwards Method · Computational Efficiency
Explores the sociological history of neural networks and the early mathematical foundations of the 'backwards method' for sensitivity analysis.
The Core Backpropagation Algorithm
Key concepts: Chain Rule · Gradient Computation · Hidden Units · Representation Learning · Stochastic Gradient Descent (SGD)
Detailed breakdown of the standard backpropagation algorithm, the chain rule, and its role in representation learning.
Theoretical Derivations and Optimal Control
Key concepts: Kelley-Bryson gradient formula · Optimal control theory · Hidden-node teaching · Multilayer Perceptron (MLP)
An advanced look at backpropagation through the lens of optimal control theory and the Kelley-Bryson formula.
Backpropagation for Specialized Architectures
Key concepts: Backpropagation through time (BPTT) · Backpropagation through structure (BPTS) · Recurrent Neural Networks (RNN) · Network Unfolding
How backpropagation is adapted for sequential and recursive data structures.
Advanced Optimization and Efficiency
Key concepts: Levenberg–Marquardt (LM) algorithm · Quasi-Hessian matrix · Jacobian matrix storage · Second-order training
Methods to improve the speed and memory efficiency of backpropagation-based training.
Source Materials
Study Backpropagation on Lykke
This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.
Add the free extensionSign in to open every section