NSYSU CSE Course: Information Theory
Textbook
- T. Cover and J. Thomas, Elements of Information Theory, John Wiley and Sons, Inc., 1991, USA, ISBN: 0-471-06259-6
Announcements
Lectures
- Entropy and Mutual Information (ps)
- entropy, relative entropy, conditional entropy
- mutual information
- KL distance
- inequalities in information theory
- convex functions and convex sets, concavity
- Jensen's inequality
- log sum inequality
- information inequality
- data processing inequality
- Fano's inequality
- AEP and Entropy Rate (ps)
- asymptotic equipartition property
- typical sets
- error-free source coding based on the typical set
- stochastic processes
- entropy rates
- time-invariant Markov chains
- hidden Markov models
- Data Compression (ps)
- classification of source coding
- Kraft inequality
- optimal codes
- compression limit
- compression and gambling
- Channel Capacity (ps)
- information channel capacity
- operational channel capacity
- channel coding theorem
- source-channel coding theorem
- Gambling and Entropy (ps)
- doubling rate
- side information
- entropy of English
- Kolmogorov Complexity (ps)
- Differential Entropy (ps) and Gaussian Channels (ps)
- definitions and examples
- AEP for continuous random variables
- joint and conditional differential entropy
- relative entropy and mutual information
- multi-variate normal distributions
- capacity of a Gaussian channel
- band-limited channels
- parallel channels
- Maximum Entropy and Spectral Estimation (ps)
- IT and Statistics (ps)
- method of types
- universal source coding
- Lempel-Ziv coding
- average length of Lempel-Ziv compression
- Rate Distortion Theory (ps)
- encoder and decoder
- distortion function
- code rate
- achievable rate-distortion pair
- rate-distortion theorem
- Network Information Theory (ps)
- broadcast channel
- multiple access channel
- distributed source coding
- capacity regions
- multi-terminal network
- IT and Stock Markets (ps)
- setting up the framework
- doubling rate
- Kuhn-Tucker conditions
- asymptotic optimality
- impact of side information
Exams