605.742.81 - Deep Neural Networks

Computer Science
Summer 2024

Description

This course provides a practical introduction to deep neural networks (DNN) with the goal to extend student’s understanding of the latest and cutting-edge technology and concepts in deep learning (DL) field. DNNs are simplified representation of neurons in the brain that are suited in complex applications, such as natural language processing (NLP), computer vision (CV), speech processing, and many other predictive models rising from non-linear and unstructured data, including text, images, video, audio. The course starts with a brief review of machine learning (ML) and neural networks (NN), including anatomy of neural networks, model evaluation techniques and feature engineering in Python with TensorFlow (TF) and Keras. It then defines and exemplifies the deep learning with convolutional neural networks (CNN), recurrent neural networks (RNN), long-short term memory (LSTM) networks with attention mechanism, generative adversarial networks (GAN) and deep reinforcement learning (DRL), and transfer learning among other key concepts. This is a hands-on course with significant Python coding requirements. Students will apply neural networks to the computer vision (CV) tasks, natural language processing (NLP) tasks, and domains with structured data. Since DL is a rapidly developing field, the course will also rely on recent seminal publications, which students may be asked to reproduce with small scale datasets as an exercise. Prerequisites: Multivariate calculus, linear algebra, probability/statistics; A neural network OR machine learning course: Examples: EN.605.647, EN.625.638, EN.525.670,EN.605.649, EN.705.601, EN.605.646. A working knowledge of Python is assumed. Prior coding experience data munging, numerical linear algebra, ML, and visualization libraries is highly recommended: Example: Python, Numpy, Pandas, ScikitLearn, Matplotlib.

Expanded Course Description

This course is taught by Oleg Melnikov (LinkedIn) and Samuel Nathanson (LinkedIn).

Instructors

Profile photo of Oleg Melnikov.

Oleg Melnikov

omelnik2@jh.edu

Profile photo of Sam Nathanson.

Sam Nathanson

samuel.nathanson@jhuapl.edu

Course Structure

Course material is divided into weekly modules in Canvas. Each module runs for a period of seven (7) days and contains the following: 

  1. Viewing material: instructor recorded videos and content related YouTube videos
  2. Reading material: assigned chapter(s) for reading and related publication(s)
  3. Discussions: related to course material (including. publications)
  4. Assignments, auto-graded quizzes on HOML and related publications.
You should regularly check the upcoming deadlines in the Canvas Calendar, recent Canvas Announcements, Discussion Board for assignment due dates. JHU Academic Calendar is here. 

All dates and times (for assignments, Zoom meetings, etc.) are in ET zone! Each module (a.k.a. "week") starts each week and lasts 1 week. The assignments are released weekly and, typically, have 2 week deadline on Sundays at 11:59 pm ET.

Course Topics

We will closely follow the topics in HOML textbook (2nd edition, not newer or older!). We will rapidly advance through HOML part I to cover classical ML, build up on terminology and Kaggle/Colab/Competition skills/grading. Weekly modules (topics):

  1. HOML Ch.1-2: Machine Learning Fundamentals
  2. HOML 3-4: Classification and Regression
  3. HOML 6-7: Decision Trees and Ensembles
  4. HOML 8-9: Dimension Reduction, Unsupervised Learning
  5. HOML 10: Artificial Neural Network (ANN) with Keras
  6. HOML 11: Training a Deep Neural Network (DNN)
  7. HOML 12: Custom Models and Training with TensorFlow (TF)
  8. HOML 13: Load and Preprocess Data with TF
  9. HOML 14: Deep Computer Vision (CV) and Convolutional Neural Networks (CNN)
  10. HOML 15: Process Sequences Using Recurrent Neural Networks (RNN) and CNNs
  11. HOML 16: NLP with RNNs and Attention
  12. HOML 17: Representation Learning and Generative Adversarial Networks (GAN)
  13. HOML 18: Reinforcement Learning
  14. HOML 19: Train and Deploy Scale
The quiz questions and material are tied to specific page numbers in HOML, 2nd edition!

Course Goals

To introduce the students to the variety of deep learning (DL) models, their construction, evaluation and application.
To enable students to tie Python programming code to mathematics/statistics in order to solve real world problems we face today and in the future.

Course Learning Outcomes (CLOs)

Textbooks

Required:

Optional (free):
  1. Deep learning, Ian Goodfellow, et. al. 2016. MIT Press
  2. Pattern recognition and machine learning. Svensén, M. and Bishop, C.M., 2007.
  3. Dive into deep learning. Zhang, A., Lipton, Z.C., Li, M. and Smola, A.J., 2021. arXiv:2106.11342.
  4. Foundations of machine learning. Mohri, M., Rostamizadeh, A. and Talwalkar, A., 2018. MIT press.
  5. An introduction to statistical learning (Vol. 112, p. 18). James, G., Witten, D., Hastie, T. and Tibshirani, R., 2013. New York: Springer.
  6. The elements of statistical learning (Vol. 1, No. 10). Friedman, J., Hastie, T. and Tibshirani, R., 2001. New York: Springer series in statistics.
  7. Applications of deep neural networks. Heaton, J., 2020. arXiv:2009.05673.
Prerequisite Material:
  1. Mathematics for machine learning. Deisenroth, M.P., Faisal, A.A. and Ong, C.S., 2020. Cambridge University Press. (Source)
  2. The Matrix Cookbook. Petersen, K.B. and Pedersen, M.S., 2008. Technical University of Denmark, 7(15), p.510. (Source)
  3. Beginner's Guide to Python and Python Tutorial. Python.org

Other Materials & Online Resources

Students need a Google Drive account (available for free with Gmail account) with a Google Colab plugin. Colab is a Jupyter Notebook-like interface, which provides a seamless access to Python, TensorFlow, visualizations and (free) CPU/GPU/TPU hardware from Google. Colab ensures identical programming environment among all participants of the course.

Note: This is an individual (not a team) assignment. Please don't post the copyrighted material in the public space outside of our course; the content takes many hundreds of hours to create and maintain. See JHU's Code of Conduct, Honor Code, and Academic Integrity Policy. Please report suspected cheating incidents to the Teaching Team.

Required Software

We use Google Colab Notebooks for programming assignments, which is pre-configured with Python 3.x and most packages/versions already installed.

Student Coursework Requirements

Final grades will be determined by the following weighting:

  1. 30%: Assignments
  2. 30%: Auto-graded quizzes
  3. 10%: Participation discussion forums, incl. publication discussions
  4. 30%: Final project and presentation (e.g. Kaggle competition).
    1. The project is done in 2 stages - interim submission is 15% and final submission is 15% of the total grade
Grading Kaggle competitions.
Private leaderboard (LB) grade: the above-baseline grade is proportional to their rank within 80-100% range. The below-baseline grade is 70%. Missing participation is zero points. The LB grade is then adjusted up or down (a few points) for the modeling, reproducibility of the private LB score, documentation, references, team in LB, runtime speed, Colab organization, late penalties, Colab errors, use of non-student section in Colab and otherwise not following the competition rules. See rules in Kaggle. Thus, top LB scores can significantly drop in grades due to not following the rules, while low-performing submissions can receive a significantly higher grade. Colab submissions are peer-reviewed after the end of each competition.

Grading Policy

Score Range≥97≥94≥90≥87≥83≥80≥77≥73≥70≥67≥63≥0
Letter Grade A+AA-B+BB-C+CC-D+DF

Course Evaluation

Regrading:

  1. We aim to grade fairly, accurately, and timely. If you believe we made a crude grading error, make a private post in our discussion forum ASAP (within 1 week of grading) & tag a grader.
  2. To discourage frivolous appeals, we reserve the right to deduct a 2-5% of the grade, if your appeal lacks a strong justification or the benefit fails to exceed 2-5%. Be sure it is worth the mutual effort.
  3. If in doubt, always use the course textbook as the main source of truth. It has been well edited, provides full context, and has very few typos.
Plagiarism is a serious offense. Always reference your sources of code and ideas. We encourage learning from StockOverflow, GitHub, Kaggle Discussions, etc., but the key solution and contribution must be your own (and be clearly demonstrated).
  1. Rule 0: If ever in doubt, just ask!
  2. Every output (plot, table, metrics, ...) must have a purpose and an interpretation.
    1. We never produce outputs for the sake of filling the void.
  3. Output must be legible, readable, meaningfully organized, neat, clean, compact, precise, and concise.
  4. Avoid generic terms, such as data and model. Instead, be specific and use observations, features, linear regression, logistic regression, etc.
  5. No overplotting. No redundant plots. No data dumps (pages of table outputs).
  6. Annotate your programming code! Help your audience to understand your abstraction.
  7. All code must execute in Google Colab and Python 3.x without error/warning messages or local file dependencies.
  8. Always start your modeling with a simple baseline model (version 0), which you can do with minimal data munging.
    1. This is your Proof of Concept and benchmark for any future improvement.
  9. Start your work early! Never procrastinate. We are busy on assignment due days and may be inaccessible to answer last minute questions.
  10. Clearly label problems and order them as assigned (to help us avoid missing your solutions)
  11. IF late submissions are allowed, there will be 10% penalty per day. (Start early, manage your time, expect to get stuck!)
  12. Quiz:
    1. Students are given 2 quiz attempts with the highest grade recorded.
    2. Multi-choice questions are multi-answer
    3. Incorrect response lowers your score. Correct response raises your score (to avoid guessing).
  13. Grade disputes should be raised within 1 week of grade release. Please carefully monitor your grades in Canvas Gradebook. For example, Canvas may auto-apply last submission's late penalty to the highest grade for the assignment. We will not honor the correction, if it's noted outside of grade dispute interval.
Participation.
To encourage participation, students should provide a Peer Evaluation for the team work of their peers. Aggregated evaluation can be used to adjust the assignment and course grade. For example, we used these evaluations in the past to raise a course grade for students with good evals and leave the grades as they are for students with poor evals. A URL Link to the Google form is included in each week's material.

Academic Policies

Deadlines for Adding, Dropping and Withdrawing from Courses

Students may add a course up to one week after the start of the term for that particular course. Students may drop courses according to the drop deadlines outlined in the EP academic calendar (https://ep.jhu.edu/student-services/academic-calendar/). Between the 6th week of the class and prior to the final withdrawal deadline, a student may withdraw from a course with a W on their academic record. A record of the course will remain on the academic record with a W appearing in the grade column to indicate that the student registered and withdrew from the course.

Academic Misconduct Policy

All students are required to read, know, and comply with the Johns Hopkins University Krieger School of Arts and Sciences (KSAS) / Whiting School of Engineering (WSE) Procedures for Handling Allegations of Misconduct by Full-Time and Part-Time Graduate Students.

This policy prohibits academic misconduct, including but not limited to the following: cheating or facilitating cheating; plagiarism; reuse of assignments; unauthorized collaboration; alteration of graded assignments; and unfair competition. Course materials (old assignments, texts, or examinations, etc.) should not be shared unless authorized by the course instructor. Any questions related to this policy should be directed to EP’s academic integrity officer at ep-academic-integrity@jhu.edu.

Students with Disabilities - Accommodations and Accessibility

Johns Hopkins University values diversity and inclusion. We are committed to providing welcoming, equitable, and accessible educational experiences for all students. Students with disabilities (including those with psychological conditions, medical conditions and temporary disabilities) can request accommodations for this course by providing an Accommodation Letter issued by Student Disability Services (SDS). Please request accommodations for this course as early as possible to provide time for effective communication and arrangements.

For further information or to start the process of requesting accommodations, please contact Student Disability Services at Engineering for Professionals, ep-disability-svcs@jhu.edu.

Student Conduct Code

The fundamental purpose of the JHU regulation of student conduct is to promote and to protect the health, safety, welfare, property, and rights of all members of the University community as well as to promote the orderly operation of the University and to safeguard its property and facilities. As members of the University community, students accept certain responsibilities which support the educational mission and create an environment in which all students are afforded the same opportunity to succeed academically. 

For a full description of the code please visit the following website: https://studentaffairs.jhu.edu/policies-guidelines/student-code/

Classroom Climate

JHU is committed to creating a classroom environment that values the diversity of experiences and perspectives that all students bring. Everyone has the right to be treated with dignity and respect. Fostering an inclusive climate is important. Research and experience show that students who interact with peers who are different from themselves learn new things and experience tangible educational outcomes. At no time in this learning process should someone be singled out or treated unequally on the basis of any seen or unseen part of their identity. 
 
If you have concerns in this course about harassment, discrimination, or any unequal treatment, or if you seek accommodations or resources, please reach out to the course instructor directly. Reporting will never impact your course grade. You may also share concerns with your program chair, the Assistant Dean for Diversity and Inclusion, or the Office of Institutional Equity. In handling reports, people will protect your privacy as much as possible, but faculty and staff are required to officially report information for some cases (e.g. sexual harassment).

Course Auditing

When a student enrolls in an EP course with “audit” status, the student must reach an understanding with the instructor as to what is required to earn the “audit.” If the student does not meet those expectations, the instructor must notify the EP Registration Team [EP-Registration@exchange.johnshopkins.edu] in order for the student to be retroactively dropped or withdrawn from the course (depending on when the "audit" was requested and in accordance with EP registration deadlines). All lecture content will remain accessible to auditing students, but access to all other course material is left to the discretion of the instructor.