Home | Research | Papers | Talks | Teaching |

This is an introductory course on information theory.
I will be covering most of chapters 2-5, 9-10
and a few additional topics from chapters 10,15
of the book Elements of Information Theory (2nd ed.) by Cover and Thomas.

Prerequisites: EE3110 or an equivalent course on probability.

See here for a previous offering of this course with a different course number.

Textbook: Elements of Information Theory (2nd ed.) T. M. Cover and J. A. Thomas.

Course topics.

- Introduction to information theory
- Entropy, relative entropy and mutual information
- Asymptotic equipartition property
- Entropy rate of a stochastic process
- Data compression
- Channel capacity
- Differential entropy
- Gaussian channel
- Additional topics (time permitting)

Grading policy (Tentative)

5% Homework+scribing, 5-10% Project, 10% Miniquizzes, 25-30% Midsem, 50% Endsem.

Midsem 01 Oct 2019

Endsem 16 Nov 2019 (as per Institute calender)

Lectures

- 29 Jul Lecture 01 Introduction

- 30 Jul Lecture 02 Entropy, joint and conditional entropies

- 05 Aug Lecture 03 Relative entropy, mutual information

- 06 Aug Lecture 04 Jensen's inequality and consequences

- 13 Aug Lecture 05 Log sum inequality and consequences

- 19 Aug Lecture 06 Data processing inequality, Fano's inequality

- 20 Aug Tutorial

- 22 Aug Lecture 07 AEP, Miniquiz-1

- 27 Aug Lecture 08 AEP, Entropy rate

- 29 Aug Lecture 09 Entropy rates of stochastic processes

- 03 Sep Lecture 10 Data compression: Source coding, Kraft's inequality

- 05 Sep Lecture 11 Extended Kraft's inequality, bounds

- 09 Sep Tutorial

- 10 Sep Lecture 12 Feb Uniquely decodable codes, Introduction to Huffman codes

- 19 Sep Lecture 13 Huffman codes, optimality of Huffman codes

- 23 Sep Lecture 14 Channel capacities

- 24 Sep Tutorial

- 26 Sep Lecture 15 Joint AEP

- 01 Oct Midsem exam

- 07 Oct Noninstructional day

- 08 Oct Dussehra holiday

- 10 Oct Lecture 16 Channel coding theorem

- 17 Oct Lecture 17 Converse of channel coding theorem, source channel separation

- 21 Oct Lecture 18 Differential entropy

- 22 Oct Lecture 19 Differential entropy, Gaussian channel

- 28 Oct Lecture 20 Capacity of Gaussian channel

- 29 Oct Lecture 21 Parallel and bandlimited Gaussian channels

- 31 Oct Tutorial, Miniquiz-3