ECE 547 / CS 591.04: Neural Networks

Instructors

• Professor: Barak Pearlmutter <bap@cs.unm.edu>
  Office: FEC 355C, Phone: 277-9425
  office hours: Tue/Thur 10:00-12:00

  

• TA: Douglas Eck <deck@cs.unm.edu>
  Office: FEC 355, Phone: 277-9786
  office hours: Mon/Wed 10:00-12:00

Course Materials

• Time: Fall 1998, Tue/Thur 12:30-1:45

• Room: CE 103

  

• Textbook: Neural Networks: a Comprehensive Foundation, Simon Haykin, 2nd ed, 1999, Prentice Hall.

• Supplementary text: Reinforcement Learning: An Introduction, Richard S. Sutton and Andrew G. Barto, 1998, MIT Press. (Both are in the bookstore, on the CS shelf rather than the EECE shelf. Amazon.com also stocks both texts.)

  
  
  

• Here are the links to RL techniques promised over and over again in lecture
  • An on-line reinforcement learning tutorial
  • Reinforcement Learning Repository at Michigan State University
  • Sutton & Barto Book: Reinforcement Learning: An Introduction (A duplicate link since this is a supplementary text.)

• Many assignments will use the MNIST Database of Handwritten Digits, a version of the MNIST digits database massaged by Yann LeCun. (A copy of the dataset will be available on the course machines.)

• There is some code available for the class. Copies are also in /usr/local/pkg/ece547/ on the NIIEL and CIRT ESC Pod Linux clusters.
  • plotonline, a Linux program that will show a constantly-updated plot of a series of numbers (eg error rates) as they are generated, or a constantly-updated histogram.
  • Code for reading MNIST data, for writing PPM files, and for converting a bunch of PPM files to an animated GIF.
  As the course progresses, we will add more code.

Computing Resources

• On-campus linux boxes
  For those who need to sit at a machine on campus, use the CIRT Escape Pod. There are dual-boot linux/NT machines there.
• Cycle servers
  For off-campus use (or terminal access from anywhere...) log into the NIIEL machines. They are named niielXX.cs.unm.edu where XX is a two-digit number between 01 and 20.
• Comments
  • Both environments should be more or less identical. You should add /usr/local/pkg/ece547 to your path. We've put the MNIST database and all of the helper code there already.
  • The NIIEL machines are only available via ssh. See this site for a ssh client for Windows 95/NT
  • The NIIEL machines require special accounts which will be given out in class as soon as they are created.
  • The CIRT ESC Pod machines use your CIRT login account
  • There will certainly be small problems for the first two or three assignments. Please be patient as we work out the bugs!

Syllabus

Everyone in the class will do a final project (typically in teams of two). Neural networks are my research area, and I am very happy to help students who become particularly excited about some topic learn more about it.

Course materials including assignments will be posted on the web. (You can get extra credit by writing up html lecture notes in a timely fashion for me to post on the web.)

Topics to be covered

• The nervous system and how neural networks emerged from the ooze
• kinds of learning: supervised, unsupervised, reinforcement
• perceptrons
• backward propagation of error and multilayer perceptions
• competitive learning
• The EM algorithm
• principle components analysis
• information theoretic models
• independent components analysis
• Boltzmann machines (and simulated annealing and the Kulback-Liebler distance and the maximum likelihood criterion)
• Helmholtz machines
• fixedpoint networks, mean-field networks
• recurrent networks
• reinforcement learning using ARP units, policy/value iteration, Q-learning

Prerequisites

• have taken differential calculus (eg know the chain rule, know what a gradient is.)
• be able to program a computer.
• know a little linear algebra (eg know what the inverse of a matrix is, have some idea about eigenvalues.)
• have some idea what a first-order differential equation is.
• know or be able to quickly pick up what a probability distribution is.

I'm not a stickler about formal prereqs - effort can make up for a lot. But I expect you to not be scared of working through equations when they come up, or thinking in mathematical terms. Let's say ``mathematical training or appropriate mathematical maturity.''

Schedule

The book is much more signal-processing oriented than this course. You are not obligated to understand material unrelated to the lectures.

Tue	Aug 25	historical background
Thu	Aug 27	my idiosyncratic view of neural networks (ch 2)
Tue	Sep 1	perceptrons (ch 3) assignment
Thu	Sep 3	in-class demo of perceptron learning
Tue	Sep 8	perceptrons: histograms, Widrow's LMS, multiplicative updates
Thu	Sep 10	perceptrons: theoretical limitations, randomization
Tue	Sep 15	multilayer perceptrons and backpropagation of error (ch 4)
Thu	Sep 17	more backpropagation, assignment
Tue	Sep 22	Kolmogorov's theorem; debugging gradient optimization
Thu	Sep 24	autoassociative networks and dimmensionality reduction
Tue	Sep 29	overfitting and generalization
Thu	Oct 1	more generalization
Tue	Oct 6	VC dimension, assignment
Thu	Oct 8	PAC learning, VC dimension of MLPs
Tue	Oct 13	convergence rate of gradient descent
Thu	Oct 15	(Fall break)
Tue	Oct 20	The Bayesian approach to generalization
Thu	Oct 22	(continued) assignment
Tue	Oct 27	Gaussian mixture models, EM, LVQ
Thu	Oct 29	Hierarchical Mixtures of Experts
Tue	Nov 3	Boltzmann machines
Thu	Nov 5	Fixedpoint backpropagation, MFT Boltzmann take-home quiz
Tue	Nov 10	(Doug Eck) Reinforcement learning
Thu	Nov 12	(Doug Eck) (continued)
Tue	Nov 17	(Doug Eck) Q-learning
Thu	Nov 19	Helmholtz machines
Tue	Nov 24	Blind source separation
Thu	Nov 26	(Thanksgiving)
Tue	Dec 1	(Michael Zibulevsky) Support Vector Machines
Thu	Dec 3	(Doug Eck) Recurrent Nets
Tue	Dec 8	Backpropagation through time, RTRL
Thu	Dec 10	final project presentations
Thu	Dec 17	5pm, final exam due