Introduction to Information Theory

Course requirements: Examination and grading: Grades will be based on a final exam SSD: Information Engineering Aim: The aim of this course is to introduce basic information theoretic concepts to students. We will start by introducing entropy, divergence, and, mutual information, and their mathematical properties. The rest of the course will be dedicated to illustrating engineering applications of these seemingly abstract quantities. We will see that entropy corresponds to the ultimate limit in data compression, divergence provides the best error exponent in hypothesis testing (i.e., binary classification), and mutual information sets the limit of how much data one can transmit reliably over a noisy communication channel. Syllabus: 1. Week 1: Information measures • Day 1: Entropy, divergence, mutual information • Day 2: Properties of information measures (chain rule, data processing inequality, convexity) 2. Week 2: Lossless data compression • Day 1: Asymptotic equipartition property (AEP) • Day 2: Kraft inequality, Huffman coding and its optimality 3. Week 3: Information theory and learning • Day 1: Method of types, universal source coding, large deviations: Sanovs theorem • Day 2: Hypothesis testing, Steins lemma, Chernoff exponent 4. Week 4: Channel coding • Day 1: Channel capacity theorem, achievability, joint AEP • Day 2: Converse to channel coding theorem, feedback capacity, Joint source-channel coding