525.776.81 - Information Theory

Course Structure

The course materials are divided into modules which can be accessed by clicking Modules on the left menu. A module will have several sections including the overview, content, readings, discussions, and assignments. Each module has one or more discussion forums related to the lecture material as well as the homework assignments. You are encouraged to preview all sections of the module before starting. All modules run for a period of seven (7) days, and detailed dates are noted in the Course Outline. Students should regularly check the Calendar and Announcements for assignment due dates.

Course Topics

• Introduction to Information Theory
• Information & Entropy
• Chain Rules
• Jensen’s Inequality & Consequences
• Asymptotic Equipartition Property
• Stochastic Processes & Entropy Rate
• Data Compression & Source Coding Theorem
• Huffman Codes
• Arithmetic Codes
• Channel Capacity
• Error-Correcting Codes
• Differential Entropy
• Communication Over Gaussian Channels

Course Goals

The primary goal of this course is to explain to students how information is measured in terms of entropy, relative entropy, mutual information. Furthermore, this course will demonstrate how these quantities are used to derive fundamental limits to data compressibility and communication rates over a noisy channel. Finally, students will be exposed to optimal coding schemes for compressing and transmitting data.

Course Learning Outcomes (CLOs)

• Define the fundamental concepts of entropy, relative entropy, and mutual information in terms of probability and random variables.
• Explain how entropy and related quantities can be used to describe the information content of real-world data source(s).
• Implement basic data compression schemes using principles derived from information- theoretic concepts.
• Apply information-theoretic coding schemes for efficiently transmitting data over a noisy channel.
• Examine how information-theoretic concepts are employed in state-of-the-art coding schemes.
• Determine other potential application areas for information theory, such as gambling and investment strategy, big data analytics, pattern recognition and machine learning, robotics and autonomous control, etc.

Textbooks

Cover, T.M. & Thomas, J.A. (2006). Elements of information theory (2nd ed.). Hoboken, NJ: John Wiley & Sons, Inc.

ISBN-13: 978-0-471-24195-9

MacKay, D.J.C. (2003). Information theory, inference, and learning algorithms. Cambridge, UK: Cambridge University Press.

ISBN-13 978-0-521-64298-9.

Required Software

Students will need access to a recent version of MATLAB with the Statistics Toolbox. The MATLAB Total Academic Headcount (TAH) license is now in effect. This license is provided at no cost to you. Send an email to software@jhu.edu to request your license file/code. Please indicate that you need a standalone file/code. You will need to provide your first and last name, as well as your Hopkins email address. You will receive an email from Mathworks with instructions to create a Mathworks account. The MATLAB software will be available for download from the Mathworks site.

Student Coursework Requirements

It is expected that each module will take approximately 7–10 hours per week to complete. Here is an approximate breakdown: reading the assigned sections of the texts (approximately 1–2 hours per week), listening to the audio annotated slide presentations (approximately 2 hours per week), homework assignments (approximately 5 hours per week), and additional time spent working on group and individual research projects.

This course will consist of five basic student requirements:

Preparation and Participation (10% of Final Grade Calculation)

Students are responsible for carefully reading all assigned material and being prepared for discussion. The majority of readings are from the course text. Additional reading may be assigned to supplement text readings.

Discussion forums are included with each module and participation in the forums will be counted towards the final grade. Most of the modules have a discussion forum that begin with a topic or question initiated by me. All of the modules have a general forum for questions and comments relevant to the lectures for that module. Finally, each module’s assignment has a collaboration forum with separate threads for each problem assigned.

Students should post an initial response to the discussion questions by the evening of day 5 for that module week and respond to classmate postings by the evening of day 8. Just posting a response to a discussion question is not sufficient; we want each student to interact with their classmates. Posting responses to the discussion question and classmate postings constitute part one of the grade for module discussions (i.e., Timeliness).

Part two of the grade for module discussion is the content, insight, and analyses in the posting and interaction (i.e., Critical Thinking). Be detailed in your postings and in your responses to your classmates' postings. Feel free to agree or disagree with your classmates. Please ensure that all postings are civil and constructive.

I will monitor module discussions and will respond to some of the discussions as discussions are posted. In some instances, I will summarize the overall discussions and post the summary for the module.

Evaluation of preparation and participation is based on contribution to discussions.

Preparation and participation is evaluated by the following grading elements:

1. Timeliness (50%)
2. Critical Thinking (50%)

Preparation and participation is graded as follows:

• 100–90 = A—Timeliness [regularly participates; all required postings; early in discussion; throughout the discussion]; Critical Thinking [rich in content; full of thoughts, insight, and analysis].
• 89–80 = B—Timeliness [frequently participates; all required postings; some not in time for others to read and respond]; Critical Thinking [substantial information; thought, insight, and analysis has taken place].
• 79–70 = C—Timeliness [infrequently participates; all required postings; most at the last minute without allowing for response time]; Critical Thinking [generally competent; information is thin and commonplace].
• <70 = F—Timeliness [rarely participates; some, or all required postings missing]; Critical Thinking [rudimentary and superficial; no analysis or insight is displayed].

Homework Assignments (20% of Final Grade Calculation)

Assignments will mostly include quantitative problem sets relevant to the topics covered in the module. Some assignments will consist of MATLAB exercises. Unless the assignment is specifically called out as a MATLAB exercise, students are expected to complete the assignment without use of any special software.

Instructions for non-MATLAB assignments: Although students are allowed to collaborate with other students via the discussion forums, they must submit their answers individually and show all work to receive full credit for each problem. Students may submit either handwritten or typed solutions. Handwritten submissions should be written legibly in blue or black ink, scanned, and submitted in PDF format. Typed solutions should be prepared using software designed for mathematical typesetting only. Students are encouraged to use LaTeX, but solutions prepared in MS-Word will be accepted only if the equation editor (or a third-party plugin, such as MathType) is used for mathematical expressions. Students submitting typed solutions must submit the typeset PDF as well as the “native” format, i.e. .tex or .docx files. All fonts necessary for viewing the completed assignment must be embedded in the submitted files. Submissions created by using software with built-in analytical solvers, such as Maple or Mathematica, will not be accepted.

Instructions for MATLAB assignments: Students must submit all code (m-files) written to complete the assignment. Code must be commented thoroughly and include a header that describes how to use it. I will grade the assignment by running the code according to the description in the header. It is important that students test their code thoroughly to ensure that it does not require any special auxiliary files or toolboxes (aside from basic ones, like the MATLAB Statistics Toolbox). If you are not a MATLAB user and would like to complete the assignment in a different programming language, I am willing to work out alternatives on a case-by-case basis.

All assignments are due according to the dates in the Calendar and Course Outline.

Late submissions will be accepted until one week past the deadline, but will be reduced by one letter grade (no exceptions without prior coordination with me). I will post solutions to the homework assignment, for your reference, after the late submission window has passed.

Homework assignments are evaluated by the following grading elements:

• Each part of question is answered (20%)
• Assumptions are clearly stated (20%)
• Intermediate derivations and calculations are provided (25%)
• Answer is technically correct and is clearly indicated (25%)
• Answer precision and units are appropriate (10%)

Homework assignments are graded as follows:

• 100–90 = A—All parts of question are addressed; All assumptions are clearly stated; All intermediate derivations and calculations are provided; Answer is technically correct and is clearly indicated; Answer precision and units are appropriate.
• 89–80 = B—All parts of the question are addressed; All assumptions are clearly stated; Some intermediate derivations and calculations are provided; Answer is technically correct and is indicated; Answer precision and units are appropriate.
• 79–70=C—Majority of parts of the question are addressed; Assumptions are partially stated; Few intermediate derivations and calculations are provided; Answer is not technically correct but is indicated; Answer precision and units are indicated but inappropriate.
• <70=F—Some parts of the question are addressed; Assumptions are not stated; Intermediate derivations and calculations are not provided; The answer is incorrect or missing; The answer precision and units are inappropriate or missing.

Group Project (20% of Final Grade Calculation)

A group project will be assigned several weeks into the course. Groups of students determined by the instructor will be assigned an advanced topic in information theory to present to the class. Module 10 will be devoted to presentations of group projects.

The Group project is evaluated by the following grading elements:

• Accuracy and completeness of the team’s presentation of the technical content This portion of the grade will be the same for each member of the team. (30%).
• Clarity and effectiveness of the team’s explanation of the subject to the class. This portion of the grade will be the same for each member of the team (20%).
• Participation in answering questions and facilitating discussion of their team’s topic. This portion of the grade may be different for each member of the team (20%).
• Each student’s participation in discussion forums for other teams’ topics. This portion of the grade may be different for each member of the team (20%).
• Peer evaluation of each student’s contributions to their team’s project. This portion of the grade may be different for each member of the team (10%).

The Group Project is graded as follows:

• 100–90 = A—Student Preparation and Participation/ Team Preparation and Participation [individual/ team roles and responsibilities well defined and understood; individual/ team well versed in use of Adobe Connect; individual/ team work product(s) agreed to, well prepared and available to all team members/ instructors]; Student Understanding/ Team Understanding [rich in content; full of thought, insight, and analysis].
• 89–80 = B—Student Preparation and Participation/ Team Preparation and Participation [individual/ team roles and responsibilities well defined and understood; individual/ team well versed in use of Adobe Connect; individual/ team work product(s) agreed to and prepared]; Student Understanding/ Team Understanding [substantial information; thought, insight, and analysis has taken place].
• 79–70 = C—Student Preparation and Participation/ Team Preparation and Participation [individual/ team roles and responsibilities agreed to; individual/ team well versed in use of Adobe Connect; individual/ team work product(s) prepared]; Student Understanding/ Team Understanding [generally competent; information is thin and commonplace].
• <70 = F—Student Preparation and Participation/ Team Preparation and Participation [individual/ team roles and responsibilities not well understood; individual/ team has difficult with use of Adobe Connect; individual/ team work product(s) partially prepared]; Student Understanding/ Team Understanding [rudimentary and superficial; no analysis or insight displayed].

Individual Research Project (20% of Final Grade Calculation)

An individual research project will be assigned several weeks into the course. It will be each student’s responsibility to choose a topic that is relevant to the course and may be investigated appropriately in the given time frame. Deadlines for submitting project titles, 500-word abstracts, and posters in PowerPoint format are given in the Calendar and Course Outline. Module 13 will include a “virtual poster session” consisting of discussion forums for each student’s poster.

The individual project is evaluated by the following grading elements:

• Submission of project title, abstract, and poster by the specified deadlines (10%).
• Accuracy and completeness of the poster’s technical content (30%).
• Clarity of the abstract in summarizing the work and its significance, effectiveness of the poster in illustrating quantitative and qualitative results (20%).
• Participation in answering questions and facilitating discussion of their own poster (20%).
• Participation in discussion forums for other students’ posters. (20%).

The individual project is graded as follows:

• 100–90 = A—Student Preparation and Participation [title, abstract, and poster submitted on time; engages the class in lively discussion; provides seedling questions for discussion on own work]; Student Understanding [topic, motivation, and objectives are explicitly defined; abstract is concise yet informative; poster summarizes past work; explains methodology; concisely illustrates quantitative and qualitative results; care is taken with regard to data visualization; provides thorough analysis].
• 89–80 = B— Student Preparation and Participation [title, abstract, and poster submitted on time; asks insightful questions regarding each students’ topics; sufficiently answers each student’s questions regarding their own topic]; Student Understanding [topic, motivation, and objectives are broadly outlined; abstract summarizes the project accurately; poster summarizes past work; explains methodology; shows qualitative or quantitative results; some analysis provided].
• 79–70 = C— Student Preparation and Participation [title, abstract, or poster submitted late; asks insightful questions regarding some students’ topics; sufficiently answers most questions regarding their own topic]; Student Understanding [topic, motivation, and objectives are vaguely outlined; abstract summarizes topic but is open-ended; poster summarizes past work; explains methodology; few results or subsequent analysis provided].
• <70 = F— Student Preparation and Participation [title, abstract, and/or poster submitted late; ignores students questions about their own topic; does not participate in discussion of other students’ topcis]; Student Understanding [topic, motivation, and/or objectives not mentioned; abstract only states the topic investigated; no literature review performed; methodology not explained; no results or analysis provided].

Exams (30% of Final Grade Calculation, combined from 15% for Midterm and 15% for Final)

The midterm exam will be available in Module 6, and will cover topics presented in Modules 1-6. The final exam will be available in Module 14, and will cover all topics presented in the course with more emphasis on Modules 7-9 and 11-14.

Exams will be due on the dates specified in the calendar and course outline. Late submissions will be accepted until two days past the deadline, and will be reduced by one letter grade. Submissions later than two days past the deadline will not be accepted.

You may use the course text, previous homework assignments, and previous course discussions to complete the exams. Discussion of the exam with other students is strictly prohibited.

Each exam will be graded out of 100 points. The point value of each question will be shown. Answers to each question will be scored by the following grading elements:

• Each part of question is answered (20%)
• Assumptions are clearly stated (20%)
• Intermediate derivations and calculations are provided (25%)
• Answer is technically correct and is clearly indicated (25%)
• Answer precision and units are appropriate (10%)

Exams are graded as follows:

• 100–90 = A—All parts of question are addressed; All assumptions are clearly stated; All intermediate derivations and calculations are provided; Answer is technically correct and is clearly indicated; Answer precision and units are appropriate.
• 89–80 = B—All parts of question are addressed; All assumptions are clearly stated; Some intermediate derivations and calculations are provided; Answer is technically correct and is indicated; Answer precision and units are appropriate.
• 79–70=C—Most parts of question are addressed; Assumptions are partially stated; Few intermediate derivations and calculations are provided; Answer is not technically correct but is indicated; Answer precision and units are indicated but inappropriate.
• <70=F—Some parts of the question are addressed; Assumptions are not stated; Intermediate derivations and calculations are not provided; The answer is incorrect or missing; The answer precision and units are inappropriate or missing.

Grading Policy

Assignments are due according to the dates posted in the course site. Students should check these due dates in the Calendar or Course Outline. I will post grades approximately one week after assignment due dates.

I generally do not directly grade spelling and grammar. However, egregious violations of the rules of the English language will be noted without comment. Consistently poor performance in either spelling or grammar is taken as an indication of poor written communication ability that may detract from your grade.

A grade of A indicates achievement of consistent excellence and distinction throughout the course—that is, conspicuous excellence in all aspects of assignments and discussion in every week.

A grade of B indicates work that meets all course requirements on a level appropriate for graduate academic work. These criteria apply to both undergraduate and graduate students taking the course.

EP uses a +/- grading system (see “Grading System”, Graduate Programs catalog, p. 10).

100-97 = A+
96-93 = A
92-90 = A−
89-87 = B+
86-83 = B
82-80 = B−
79-77 = C+
76-73 = C
72-70 = C−
69-67 = D+
66-63 = D
<63 = F

Final grades will be determined by the following weighting:
Preparation and Participation: 10%
Homework Assignments: 20%
Group Project: 20%
Individual Research Project: 20%
Exams (Midterm + Final): 30% (15% + 15%)

Academic Policies

Deadlines for Adding, Dropping and Withdrawing from Courses

Students may add a course up to one week after the start of the term for that particular course. Students may drop courses according to the drop deadlines outlined in the EP academic calendar (https://ep.jhu.edu/student-services/academic-calendar/). Between the 6th week of the class and prior to the final withdrawal deadline, a student may withdraw from a course with a W on their academic record. A record of the course will remain on the academic record with a W appearing in the grade column to indicate that the student registered and withdrew from the course.

Academic Misconduct Policy

All students are required to read, know, and comply with the Johns Hopkins University Krieger School of Arts and Sciences (KSAS) / Whiting School of Engineering (WSE) Procedures for Handling Allegations of Misconduct by Full-Time and Part-Time Graduate Students.

This policy prohibits academic misconduct, including but not limited to the following: cheating or facilitating cheating; plagiarism; reuse of assignments; unauthorized collaboration; alteration of graded assignments; and unfair competition. Course materials (old assignments, texts, or examinations, etc.) should not be shared unless authorized by the course instructor. Any questions related to this policy should be directed to EP’s academic integrity officer at ep-academic-integrity@jhu.edu.

Students with Disabilities - Accommodations and Accessibility

Johns Hopkins University values diversity and inclusion. We are committed to providing welcoming, equitable, and accessible educational experiences for all students. Students with disabilities (including those with psychological conditions, medical conditions and temporary disabilities) can request accommodations for this course by providing an Accommodation Letter issued by Student Disability Services (SDS). Please request accommodations for this course as early as possible to provide time for effective communication and arrangements.

For further information or to start the process of requesting accommodations, please contact Student Disability Services at Engineering for Professionals, ep-disability-svcs@jhu.edu.

Student Conduct Code

The fundamental purpose of the JHU regulation of student conduct is to promote and to protect the health, safety, welfare, property, and rights of all members of the University community as well as to promote the orderly operation of the University and to safeguard its property and facilities. As members of the University community, students accept certain responsibilities which support the educational mission and create an environment in which all students are afforded the same opportunity to succeed academically. 

For a full description of the code please visit the following website: https://studentaffairs.jhu.edu/policies-guidelines/student-code/

Classroom Climate

JHU is committed to creating a classroom environment that values the diversity of experiences and perspectives that all students bring. Everyone has the right to be treated with dignity and respect. Fostering an inclusive climate is important. Research and experience show that students who interact with peers who are different from themselves learn new things and experience tangible educational outcomes. At no time in this learning process should someone be singled out or treated unequally on the basis of any seen or unseen part of their identity. 
 
If you have concerns in this course about harassment, discrimination, or any unequal treatment, or if you seek accommodations or resources, please reach out to the course instructor directly. Reporting will never impact your course grade. You may also share concerns with your program chair, the Assistant Dean for Diversity and Inclusion, or the Office of Institutional Equity. In handling reports, people will protect your privacy as much as possible, but faculty and staff are required to officially report information for some cases (e.g. sexual harassment).

Course Auditing

When a student enrolls in an EP course with “audit” status, the student must reach an understanding with the instructor as to what is required to earn the “audit.” If the student does not meet those expectations, the instructor must notify the EP Registration Team [EP-Registration@exchange.johnshopkins.edu] in order for the student to be retroactively dropped or withdrawn from the course (depending on when the "audit" was requested and in accordance with EP registration deadlines). All lecture content will remain accessible to auditing students, but access to all other course material is left to the discretion of the instructor.