資訊理論與壓縮編碼的應用(英文授課)

In the near future, big data analysis will change our daily life significantly. However, in practice, it is possible that several megabytes of data may contain only a few bytes of useful information. Therefore, it is essential to understand the "nature" of information, and how to derive a mathematical model such that the amount of information in a data set can be quantified for analysis. This course is based on the classical information theory proposed by C. E. Shannon, with special focus on its application to data compression. Practical algorithms for different types of data compression will be presented. Finally, two case studies on data compression, one for deep-learning neural network models and the other one for audio data, will be investigated.

Linear Algebra, Probability

無備註

The E3 website will be used to host all class materials.

Performance evaluation is based on some programming assignments, a midterm exam, and a final project. You can use the language of your choice for the assignments.

• 1. Introduction to information theory

  The history of information theory and the concept of channel and coding.

  講授：

  3

• 2. Information measures

  Defining information measure from a probability-based viewpoint, including entropy, relative entropy and mutual Information.

  講授：

  3

• 3. Data compression and Huffman code.

  Information theory of data compression and practical algorithms for Huffman coding.

  講授：

  6

• 4. Entropy Rates

  Introduction to AEP and the entropy rate of a data source.

  講授：

  3

• 5. Kolmogorov data complexity

  Defining information measure from an algorithmic viewpoint.

  講授：

  3

• 6. Arithmetic coding and dictionary-based techniques

  More on practical data compression algorithms.

  講授：

  6

• 7. Context-based compression

  Introduction to data compression using context information.

  講授：

  3

• 8. Quantization and rate-distortion theory

  Introduction to the theory of lossy data compression.

  講授：

  4

• 9. Transform-domain and sub-band coding

  Introduction to data representation in transform domain and its application for data compression.

  講授：

  5

• 10. Case Study I: Deep-learning neural network model coding

  Introduction to neural network model compressed.

  講授：

  3

• 11. Case Study II: Audio coding

  Introduction to practical algorithms for speech and audio data compression.

  講授：

  3

1. Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, 2nd Ed., John Wiley & Sons, Inc., 2006. 2. Khalid Sayood, Introduction to Data Compression, Third Edition, Morgan Kaufmann, 2005. Note that both textbooks are available in ebook format in the NCTU university library. They are not the latest editions of the textbooks from the publishers.

地點

EC718

時間

教師未提供此項資料

聯絡方式

cjtsai@cs.nctu.edu.tw (03) 573-1628 On campus extension number: 31628