learning invariances in neural nets
(40 minutes to learn)
The human visual system is capable of recognizing objects despite changes in factors such as location, orientation, and lighting. We'd like the representations learned by neural networks to be invariant to at least some of these things as well. There are several different strategies for achieving this, including enforcing invariance in the network architecture, using an appropriate regularization term, or generating randomly perturbed training data.
This concept has the prerequisites:
Core resources (read/watch one of the following)
→ Coursera: Neural Networks for Machine Learning (2012)
An online course by Geoff Hinton, who invented many of the core ideas behind neural nets and deep learning.
- Lecture "Why object recognition is difficult"
- Lecture "Achieving viewpoint invariance"
→ Pattern Recognition and Machine Learning
A textbook for a graduate machine learning course, with a focus on Bayesian methods.
Location: Section 5.5.3, pages 261-263
- Some particular strategies for learning invariances:
- building it explicitly into the architecture, as in convolutional nets
- augmenting the training set with warped examples
- tangent propagation , which penalizes instability with respect to transformations
- Tikhonov regularization , which penalizes instability with respect to noise
- The representations in the human visual system are also invariant to various transformations.
- create concept: shift + click on graph
- change concept title: shift + click on existing concept
- link together concepts: shift + click drag from one concept to another
- remove concept from graph: click on concept then press delete/backspace
- add associated content to concept: click the small circle that appears on the node when hovering over it
- other actions: use the icons in the upper right corner to optimize the graph placement, preview the graph, or download a json representation