Backpropagation

View original course

11 study materials · 5 sections

This course provides a comprehensive exploration of Backpropagation, the fundamental algorithm for training neural networks. It covers the historical evolution of the method from its roots in sensitivity analysis and optimal control to its modern application in deep learning. Students will examine the mathematical foundations of gradient computation, its extension to recurrent and recursive structures, and advanced optimization techniques for large-scale data.

Course Sections

Historical Context and the Backwards Method

Key concepts: Interpretative Flexibility · Perceptron Controversy · Ordered Derivatives · Backwards Method · Computational Efficiency

Explores the sociological history of neural networks and the early mathematical foundations of the 'backwards method' for sensitivity analysis.

The Core Backpropagation Algorithm

Key concepts: Chain Rule · Gradient Computation · Hidden Units · Representation Learning · Stochastic Gradient Descent (SGD)

Detailed breakdown of the standard backpropagation algorithm, the chain rule, and its role in representation learning.

Theoretical Derivations and Optimal Control

Key concepts: Kelley-Bryson gradient formula · Optimal control theory · Hidden-node teaching · Multilayer Perceptron (MLP)

An advanced look at backpropagation through the lens of optimal control theory and the Kelley-Bryson formula.

Backpropagation for Specialized Architectures

Key concepts: Backpropagation through time (BPTT) · Backpropagation through structure (BPTS) · Recurrent Neural Networks (RNN) · Network Unfolding

How backpropagation is adapted for sequential and recursive data structures.

Advanced Optimization and Efficiency

Key concepts: Levenberg–Marquardt (LM) algorithm · Quasi-Hessian matrix · Jacobian matrix storage · Second-order training

Methods to improve the speed and memory efficiency of backpropagation-based training.

Source Materials

Study Backpropagation on Lykke

This course page is free to open: signed out, you can read its first two sections, and signing in, on any plan, opens the rest. The free Lykke Chrome extension syncs your own Canvas courses, so ChatGPT, Claude and Muse can answer from your classes and deadlines. It only reads Canvas.

Add the free extensionSign in to open every section

Browse more course pages