525.776.81 - Information Theory

Electrical and Computer Engineering
Spring 2024


Information theory concerns the fundamental limits for data compressibility and the rate at which data may be reliably communicated over a noisy channel. Course topics include measures of information, entropy, mutual information, Markov chains, source coding theorem, data compression, noisy channel coding theorem, error-correcting codes, and bounds on the performance of communication systems. Classroom discussion and homework assignments will emphasize fundamental concepts, and advanced topics and practical applications (e.g., industry standards, gambling/finance, machine learning) will be explored in group and individual research projects.


Default placeholder image. No profile image found for Emre Gunduzhan.

Emre Gunduzhan


Course Structure

The course materials are divided into modules which can be accessed by clicking Modules on the left menu. A module will have several sections including the overview, content, readings, discussions, and assignments. Each module has one or more discussion forums related to the lecture material as well as the homework assignments. You are encouraged to preview all sections of the module before starting. All modules run for a period of seven (7) days, and detailed dates are noted in the Course Outline. Students should regularly check the Calendar and Announcements for assignment due dates.

Course Topics

Course Goals

The primary goal of this course is to explain to students how information is measured in terms of entropy, relative entropy, mutual information. Furthermore, this course will demonstrate how these quantities are used to derive fundamental limits to data compressibility and communication rates over a noisy channel. Finally, students will be exposed to optimal coding schemes for compressing and transmitting data.

Course Learning Outcomes (CLOs)


Cover, T.M. & Thomas, J.A. (2006). Elements of information theory (2nd ed.). Hoboken, NJ: John Wiley & Sons, Inc.

ISBN-13: 978-0-471-24195-9

MacKay, D.J.C. (2003). Information theory, inference, and learning algorithms. Cambridge, UK: Cambridge University Press.

ISBN-13 978-0-521-64298-9.

Note: The MacKay text is available for free as a PDF on the author’s website, provided that you do not print it out (http://www.inference.phy.cam.ac.uk/mackay/itila/book.html).

Required Software

Students will need access to a recent version of MATLAB with the Statistics Toolbox. The MATLAB Total Academic Headcount (TAH) license is now in effect. This license is provided at no cost to you. Send an email to software@jhu.edu to request your license file/code. Please indicate that you need a standalone file/code. You will need to provide your first and last name, as well as your Hopkins email address. You will receive an email from Mathworks with instructions to create a Mathworks account. The MATLAB software will be available for download from the Mathworks site.

Student Coursework Requirements

It is expected that each module will take approximately 7–10 hours per week to complete. Here is an approximate breakdown: reading the assigned sections of the texts (approximately 1–2 hours per week), listening to the audio annotated slide presentations (approximately 2 hours per week), homework assignments (approximately 5 hours per week), and additional time spent working on group and individual research projects.

This course will consist of five basic student requirements:

Preparation and Participation (10% of Final Grade Calculation)

Students are responsible for carefully reading all assigned material and being prepared for discussion. The majority of readings are from the course text. Additional reading may be assigned to supplement text readings.

Discussion forums are included with each module and participation in the forums will be counted towards the final grade. Most of the modules have a discussion forum that begin with a topic or question initiated by me. All of the modules have a general forum for questions and comments relevant to the lectures for that module. Finally, each module’s assignment has a collaboration forum with separate threads for each problem assigned.

Students should post an initial response to the discussion questions by the evening of day 5 for that module week and respond to classmate postings by the evening of day 8. Just posting a response to a discussion question is not sufficient; we want each student to interact with their classmates. Posting responses to the discussion question and classmate postings constitute part one of the grade for module discussions (i.e., Timeliness).

Part two of the grade for module discussion is the content, insight, and analyses in the posting and interaction (i.e., Critical Thinking). Be detailed in your postings and in your responses to your classmates' postings. Feel free to agree or disagree with your classmates. Please ensure that all postings are civil and constructive.

I will monitor module discussions and will respond to some of the discussions as discussions are posted. In some instances, I will summarize the overall discussions and post the summary for the module.

Evaluation of preparation and participation is based on contribution to discussions.

Preparation and participation is evaluated by the following grading elements:

  1. Timeliness (50%)
  2. Critical Thinking (50%)

Preparation and participation is graded as follows:

Homework Assignments (20% of Final Grade Calculation)

Assignments will mostly include quantitative problem sets relevant to the topics covered in the module. Some assignments will consist of MATLAB exercises. Unless the assignment is specifically called out as a MATLAB exercise, students are expected to complete the assignment without use of any special software.

Instructions for non-MATLAB assignments: Although students are allowed to collaborate with other students via the discussion forums, they must submit their answers individually and show all work to receive full credit for each problem. Students may submit either handwritten or typed solutions. Handwritten submissions should be written legibly in blue or black ink, scanned, and submitted in PDF format. Typed solutions should be prepared using software designed for mathematical typesetting only. Students are encouraged to use LaTeX, but solutions prepared in MS-Word will be accepted only if the equation editor (or a third-party plugin, such as MathType) is used for mathematical expressions. Students submitting typed solutions must submit the typeset PDF as well as the “native” format, i.e. .tex or .docx files. All fonts necessary for viewing the completed assignment must be embedded in the submitted files. Submissions created by using software with built-in analytical solvers, such as Maple or Mathematica, will not be accepted.

Instructions for MATLAB assignments: Students must submit all code (m-files) written to complete the assignment. Code must be commented thoroughly and include a header that describes how to use it. I will grade the assignment by running the code according to the description in the header. It is important that students test their code thoroughly to ensure that it does not require any special auxiliary files or toolboxes (aside from basic ones, like the MATLAB Statistics Toolbox). If you are not a MATLAB user and would like to complete the assignment in a different programming language, I am willing to work out alternatives on a case-by-case basis.

All assignments are due according to the dates in the Calendar and Course Outline.

Late submissions will be accepted until one week past the deadline, but will be reduced by one letter grade (no exceptions without prior coordination with me). I will post solutions to the homework assignment, for your reference, after the late submission window has passed.

Homework assignments are evaluated by the following grading elements:

Homework assignments are graded as follows:

Group Project (20% of Final Grade Calculation)

A group project will be assigned several weeks into the course. Groups of students determined by the instructor will be assigned an advanced topic in information theory to present to the class. Module 10 will be devoted to presentations of group projects.

The Group project is evaluated by the following grading elements:

The Group Project is graded as follows:

Individual Research Project (20% of Final Grade Calculation)

An individual research project will be assigned several weeks into the course. It will be each student’s responsibility to choose a topic that is relevant to the course and may be investigated appropriately in the given time frame. Deadlines for submitting project titles, 500-word abstracts, and posters in PowerPoint format are given in the Calendar and Course Outline. Module 13 will include a “virtual poster session” consisting of discussion forums for each student’s poster.

The individual project is evaluated by the following grading elements:

The individual project is graded as follows:

Exams (30% of Final Grade Calculation, combined from 15% for Midterm and 15% for Final)

The midterm exam will be available in Module 6, and will cover topics presented in Modules 1-6. The final exam will be available in Module 14, and will cover all topics presented in the course with more emphasis on Modules 7-9 and 11-14.

Exams will be due on the dates specified in the calendar and course outline. Late submissions will be accepted until two days past the deadline, and will be reduced by one letter grade. Submissions later than two days past the deadline will not be accepted.

You may use the course text, previous homework assignments, and previous course discussions to complete the exams. Discussion of the exam with other students is strictly prohibited.

Each exam will be graded out of 100 points. The point value of each question will be shown. Answers to each question will be scored by the following grading elements:

Exams are graded as follows:

Grading Policy

Assignments are due according to the dates posted in the course site. Students should check these due dates in the Calendar or Course Outline. I will post grades approximately one week after assignment due dates.

I generally do not directly grade spelling and grammar. However, egregious violations of the rules of the English language will be noted without comment. Consistently poor performance in either spelling or grammar is taken as an indication of poor written communication ability that may detract from your grade.

A grade of A indicates achievement of consistent excellence and distinction throughout the course—that is, conspicuous excellence in all aspects of assignments and discussion in every week.

A grade of B indicates work that meets all course requirements on a level appropriate for graduate academic work. These criteria apply to both undergraduate and graduate students taking the course.

EP uses a +/- grading system (see “Grading System”, Graduate Programs catalog, p. 10).

100-97 = A+
96-93 = A
92-90 = A−
89-87 = B+
86-83 = B
82-80 = B−
79-77 = C+
76-73 = C
72-70 = C−
69-67 = D+
66-63 = D
<63 = F

Final grades will be determined by the following weighting:
Preparation and Participation: 10%
Homework Assignments: 20%
Group Project: 20%
Individual Research Project: 20%
Exams (Midterm + Final): 30% (15% + 15%)

Academic Policies

Deadlines for Adding, Dropping and Withdrawing from Courses

Students may add a course up to one week after the start of the term for that particular course. Students may drop courses according to the drop deadlines outlined in the EP academic calendar (https://ep.jhu.edu/student-services/academic-calendar/). Between the 6th week of the class and prior to the final withdrawal deadline, a student may withdraw from a course with a W on their academic record. A record of the course will remain on the academic record with a W appearing in the grade column to indicate that the student registered and withdrew from the course.

Academic Misconduct Policy

All students are required to read, know, and comply with the Johns Hopkins University Krieger School of Arts and Sciences (KSAS) / Whiting School of Engineering (WSE) Procedures for Handling Allegations of Misconduct by Full-Time and Part-Time Graduate Students.

This policy prohibits academic misconduct, including but not limited to the following: cheating or facilitating cheating; plagiarism; reuse of assignments; unauthorized collaboration; alteration of graded assignments; and unfair competition. Course materials (old assignments, texts, or examinations, etc.) should not be shared unless authorized by the course instructor. Any questions related to this policy should be directed to EP’s academic integrity officer at ep-academic-integrity@jhu.edu.

Students with Disabilities - Accommodations and Accessibility

Johns Hopkins University values diversity and inclusion. We are committed to providing welcoming, equitable, and accessible educational experiences for all students. Students with disabilities (including those with psychological conditions, medical conditions and temporary disabilities) can request accommodations for this course by providing an Accommodation Letter issued by Student Disability Services (SDS). Please request accommodations for this course as early as possible to provide time for effective communication and arrangements.

For further information or to start the process of requesting accommodations, please contact Student Disability Services at Engineering for Professionals, ep-disability-svcs@jhu.edu.

Student Conduct Code

The fundamental purpose of the JHU regulation of student conduct is to promote and to protect the health, safety, welfare, property, and rights of all members of the University community as well as to promote the orderly operation of the University and to safeguard its property and facilities. As members of the University community, students accept certain responsibilities which support the educational mission and create an environment in which all students are afforded the same opportunity to succeed academically. 

For a full description of the code please visit the following website: https://studentaffairs.jhu.edu/policies-guidelines/student-code/

Classroom Climate

JHU is committed to creating a classroom environment that values the diversity of experiences and perspectives that all students bring. Everyone has the right to be treated with dignity and respect. Fostering an inclusive climate is important. Research and experience show that students who interact with peers who are different from themselves learn new things and experience tangible educational outcomes. At no time in this learning process should someone be singled out or treated unequally on the basis of any seen or unseen part of their identity. 
If you have concerns in this course about harassment, discrimination, or any unequal treatment, or if you seek accommodations or resources, please reach out to the course instructor directly. Reporting will never impact your course grade. You may also share concerns with your program chair, the Assistant Dean for Diversity and Inclusion, or the Office of Institutional Equity. In handling reports, people will protect your privacy as much as possible, but faculty and staff are required to officially report information for some cases (e.g. sexual harassment).

Course Auditing

When a student enrolls in an EP course with “audit” status, the student must reach an understanding with the instructor as to what is required to earn the “audit.” If the student does not meet those expectations, the instructor must notify the EP Registration Team [EP-Registration@exchange.johnshopkins.edu] in order for the student to be retroactively dropped or withdrawn from the course (depending on when the "audit" was requested and in accordance with EP registration deadlines). All lecture content will remain accessible to auditing students, but access to all other course material is left to the discretion of the instructor.