Neural Networks
0%
Course Title: Neural Networks
Course No: CSC383
Nature of the Course: Theory + Lab
Semester: 6
Full Marks: 60 + 20 + 20
Pass Marks: 24 + 8 + 8
Credit Hours: 3
Course Description
Course Objectives
Course Contents
1.1. Neural Network Fundamentals
- Basics of neural networks and human brain
- Models of a neuron
- Neural Network viewed as Directed Graphs
- Feedback
- Network Architectures
1.2. Learning and Knowledge Representation
- Knowledge Representation
- Learning Processes
- Learning Tasks
2.1. Perceptron Concepts
- Introduction
- Perceptron
- The Perceptron Convergence Theorem
2.2. Perceptron Algorithms and Classification
- Relation between the Perceptron and Bayes Classifier for a Gaussian Environment
- The Batch Perceptron Algorithm
3.1. Linear Regression and Estimation
- Introduction
- Linear Regression Model: Preliminary Considerations
- Maximum a Posteriori Estimation of the Parameter Vector
- Relationship Between Regularized Least-Squares Estimation and Map Estimation
3.2. Advanced Regression Techniques
- Computer Experiment: Pattern Classification
- The Minimum-Description-Length Principle
- Finite Sample-Size Considerations
- The instrumental-Variables Method
4.1. LMS Algorithm Fundamentals
- Introduction
- Filtering Structure of the LMS Algorithm
- Unconstrained Optimization: A Review
- The Wiener Filter
- The Least-Mean-Square Algorithm
4.2. LMS Algorithm Theory and Analysis
- Markov Model Portraying the Deviation of the LMS Algorithm from the Wiener Filter
- The Langevin Equation: Characterization of Brownian Motion
- Kushner's Direct-Averaging Method
- Statistical LMS Learning Theory for Small Learning-Rate Parameter
- Virtues and Limitations of the LMS Algorithm
- Learning-Rate Annealing Schedules
5.1. Back-Propagation Algorithm
- Introduction
- Batch Learning and On-Line Learning
- The Back-Propagation Algorithm
- XOR problem
- Heuristics for Making the back-propagation Algorithm Perform Better
- Back Propagation and Differentiation
5.2. Learning Optimization and Generalization
- The Hessian and Its Role in On-Line Learning
- Optimal Annealing and Adaptive Control of the Learning Rate
- Generalization
- Approximations of Functions
- Cross Validation
- Complexity Regularization and Network Pruning
5.3. Advanced MLP Topics
- Virtues and Limitations of Back-Propagation Learning
- Supervised Learning Viewed as Optimization Problem
- Convolutional Networks
- Nonlinear Filtering
- Small-Scale Versus Large-Scale Learning Problems
6.1. Pattern Separability and RBF Fundamentals
- Introduction
- Cover's Theorem on the separability of Patterns
- The Interpolation problem
- Radial-Basis-Function Networks
6.2. RBF Learning and Kernel Methods
- K-Means Clustering
- Recursive Least-Squares Estimation of the Weight Vector
- Hybrid Learning Procedure for RBF Networks
- Kernel Regression and Its Relation to RBF Networks
7.1. SOM Fundamentals and Properties
- Introduction
- Two Basic Feature-Mapping Models
- Self-Organizing Map
- Properties of the Feature Map
7.2. Advanced SOM Concepts
- Contextual Maps
- Hierarchical Vector Quantization
- Kernel Self-Organizing Map
- Relationship between Kernel SOM and Kullback-Leibler Divergence
8.1. RNN Architecture and Theory
- Introduction
- Recurrent Network Architectures
- Universal Approximation Theorem
- Controllability and Observability
- Computational Power of Recurrent Networks
8.2. RNN Learning Algorithms
- Learning Algorithms
- Back Propagation through Time
- Real-Time Recurrent Learning
- Vanishing Gradients in Recurrent Networks
8.3. Advanced RNN Training
- Supervised Training Framework for Recurrent Networks Using Non Sate Estimators
- Adaptivity Considerations
- Case Study: Model Reference Applied to Neurocontrol
Laboratory Works
- 1.Implement Single Layer Perceptron
- 2.Implement Multilayer Perceptron
- 3.Implement Supervised Learning algorithms
- 4.Implement Unsupervised Learning algorithms
- 5.Implement Recurrent Neural Network
- 6.Implement Linear Prediction algorithms
- 7.Implement Pattern Classification using Neural Networks
Text Books
- 1.Simon Haykin, Neural Networks and Learning Machines, 3rd Edition, Pearson
Reference Books
- 1.Christopher M. Bishop, Neural Networks for Pattern Recognition, Oxford University Press, 2003
- 2.Martin T. Hagan, Neural Network Design, 2nd Edition PWS pub co.