705.744.8VL - Deep Learning Using Transformers
Description
Transformer networks are a new trend in Deep Learning. In the last decade, transformer models dominated the world of natural language processing (NLP) and have become the conventional model in almost all NLP tasks. However, developments of transformers in computer vision were still lagging. In recent years, applications of transformers started to accelerate. This course will introduce the attention mechanism and the transformer networks by understanding the pros and cons of this architecture. The importance of unsupervised or semi-supervised pre-training for the transformer architectures, as well as their impact for developments of large-scale foundation model. This will pave the way to introduce transformers in computer vision. Additionally, the course aims to extend the attention idea into the 2D spatial domain for image datasets, investigate how convolution can be generalized using self-attention within the encoder-decoder meta architecture, analyze how this generic architecture is almost the same in image as in text and NLP, which makes transformers a generic function approximator, and discuss the channel and spatial attention, local vs. global attention among other topics. Furthermore, we will also study different neural architectures that are designed for several fundamental tasks in computer vision, namely, classification, object detection, semantic and instance segmentation. In particular, vision transformer, pyramid vision transformer, shifted window transformer (Swin), Detection Transformer (DETR), segmentation transformer (SETR), and many others will be explored. The course also examines the application of Transformers in video understanding with focus on action recognition and instance segmentation and will emphasize recent developments of transformers in large-scale pre-training and multimodal learning covering self-supervised learning, contrastive learning with masked image modeling, multimodal learning, and vision foundation models.
Expanded Course Description
Important: A computer with a recent CUDA-capable GPGPU is highly recommended. Several course topics provide examples of large neural networks, such as Large Language Models and Generative AI.
Instructor
Erhan Guven
Course Structure
The course materials are divided into modules, one for each week of the course. All course materials and assignments will be housed in Canvas and Microsoft Teams. The module content can be accessed by clicking Course Modules on the left menu. A module will have several sections including the overview, content, readings, discussions, and assignments. You are encouraged to preview all sections of the module before starting. Most modules run for a period of seven (7) days, exceptions are noted in the Course Outline. You should regularly check the Calendar and Announcements for assignment due dates.
Course Topics
- Foundations of Transformer Networks
- Large Language Models LLM Transformer Scaling Laws
- Vision Transformers ViT
- Hierarchical Vision Transformers Swin, Pyramid
- Object Detection DETR, Deformable Attention
- Semantic Segmentation SETR, MaskFormer
- Video Understanding ViViT, VisTR
- 3D and Point Cloud Understanding Point and Graph Transformers
- Contrastive Learning and Mask Image Modeling: MAE & SimMIM
- Self-Supervised Learning DINO
- Multimodal Learning CLIP
- Parameter Efficient Fine Tuning LoRA
- Robustness, Adversarial Attacks, and Safety
- Foundation Models
Course Goals
To examine the details of how transformers work and dive deep into the various designs of transformers for different benchmark tasks in computer vision. Develop theoretic understanding and hands-on experience with PyTorch-based implementations. Adopt a research perspective and expose students to state-of-the-art domain solutions using transformers.
Textbooks
Textbooks: " data-mce-style="white-space: normal;">
There is no single textbook to be used for this course. Each module contains self-contained slides, papers, or other reading materials.
Recommended
Additionally, any of the following texts or other resources that you may have from previous courses may be useful for this course if you find yourself struggling with specific skills:
- Foundations of Computer Vision. Torralba, A., Isola, P., & Freeman, W.T. ISBN-10: 0262048973.
- Modern Computer Vision with PyTorch: Explore Deep Learning Concepts and Implement Over 50 Real-World Image Applications. Ayyadevara, V. K. & Reddy, Y. ISBN-10: 1839213477.
- Deep Learning with PyTorch: Build, train, and tune neural networks using Python tools. Stevens, E., Antiga, L., & Viehmann, T. ISBN-10: 1617295264.
- Deep Learning. Goodfellow I., Bengio Y., & Courville A. ISBN-10: 0262035618.
- Official PyTorch Tutorials: https://pytorch.org/tutorials/.
EP students may access electronic versions of textbooks through the Sheridan Libraries. Instructions on how to search for available textbooks are accessible through this link: Browse Electronic Textbook Instructions
PyTorch
You will need access to a recent version of PyTorch, which is fully open-sourced and free for download.
There are several different ways to install PyTorch. We briefly provide two installation methods below. Feel free to use other installation alternatives that you may find via different sources.
- Install PyTorch to Local Machines: Depending on the operating systems of your laptops/desktops. The official PyTorch website provides different installation packages. Specific instructions can be found on https://pytorch.org/ (scroll down to the INSTALL PYTORCH section). Please note that local installation is typically time-consuming and requires lots of configurations. If you are not comfortable with such processes, proceed with other suggested installation methods.
- Use PyTorch from Google Colab Notebook: PyTorch has been built in the Colab notebook and only needs to be imported before use. This is arguably the most straightforward installation option. However, this option may suffer from limited GPU capacity that prevents users from training neural networks for longer periods. Upgrading your Google accounts may help you over these constraints with monetary prices.
- Use PyTorch on GPU servers: Some students may have access to GPU servers which natively host pre-built PyTorch images. This is a luxury and addresses all previous concerns. If one has this option, don’t hesitate to proceed with it.
All implementations (i.e., coding) are expected to be completed using Python and Google Colab Notebook (or Jupyter Notebook). For those with access to local machines or remote servers with GPUs, I strongly recommend that students modularize their code and execute via command line. Specifically, students are encouraged to develop intermediate experience with the following topics:
- Python machine learning and data science libraries, including but not limited to, NumPy, Pandas, MatplotLib, Scipy, Scikit-Learn, and others.
- Elementary single/multivariable calculus, including partial derivatives and vector calculus.
- Basic linear algebra, including matrix operations, decompositions, and eigenvectors.
- Fundamental probability and statistics, including common discrete/continuous random variables, statistical distributions, expectation, variance, sampling, estimation.
- Some machine learning basics, including training versus testing, learning taxonomy such as supervised versus unsupervised, classification versus regression, etc. Previous experience with deep learning would be helpful but not required, tensors, datasets/dataloaders, data augmentation, back-propagation, optimizers, loss functions, and fully connected neural networks etc.
Insufficient preparations in some categories may demand extra time commitment from students. If you either took these courses recently or maintained a decent recollection of roughly 70% or above on these concepts, you should be considered well-prepared. A solid PyTorch experience is extremely important.
Student Coursework Requirements
Student Coursework Requirements: " data-mce-style="white-space: normal;">
It is expected that each module will take approximately 10–16 hours per week to complete. Here is an approximate breakdown:
- Attending physical or Virtual Live classroom sessions (3 hours per week)
- Reading the assigned sections of the texts (approximately 1–2 hours per week) as well as some outside reading
- Completing assignments (approximately 4–8 hours per week)
- Researching the paper topic and engaging in weekly discussions (approximately 2–3 hours per week)
This course will consist of the following basic student requirements:
Class Participation (10% of Final Grade Calculation)You are responsible for carefully reading all assigned material and being prepared for both the Virtual Live classroom sessions and Discussions. The majority of readings are from the course text. You will be responsible for all assigned reading material, whether we cover it in class or not, so prepare questions about parts of the readings not understood. There may also be optional readings recommended from the archival literature.
Grading Policy
EP uses a +/- grading system (see “Grading System”, Graduate Programs catalog, p. 10).
| Score Range | Letter Grade |
|---|---|
| 100-98 | = A+ |
| 97-94 | = A |
| 93-90 | = A− |
| 89-87 | = B+ |
| 86-83 | = B |
| 82-80 | = B− |
| 79-77 | = C+ |
| 76-73 | = C |
| 72-70 | = C− |
| 69-67 | = D+ |
| 66-63 | = D |
| <63 | = F |
Academic Policies
Deadlines for Adding, Dropping and Withdrawing from Courses
Students may add a course up to one week after the start of the term for that particular course. Students may drop courses according to the drop deadlines outlined in the EP academic calendar (https://ep.jhu.edu/student-services/academic-calendar/). Between the 6th week of the class and prior to the final withdrawal deadline, a student may withdraw from a course with a W on their academic record. A record of the course will remain on the academic record with a W appearing in the grade column to indicate that the student registered and withdrew from the course.
Academic Misconduct Policy
All students are required to read, know, and comply with the Johns Hopkins University Krieger School of Arts and Sciences (KSAS) / Whiting School of Engineering (WSE) Procedures for Handling Allegations of Misconduct by Full-Time and Part-Time Graduate Students. This policy prohibits academic misconduct, including but not limited to the following: cheating or facilitating cheating; plagiarism; reuse of assignments; unauthorized collaboration; alteration of graded assignments; and unfair competition. Course materials (old assignments, texts, or examinations, etc.) should not be shared unless authorized by the course instructor. Any questions related to this policy should be directed to EP’s academic integrity officer at ep-academic-integrity@jhu.edu.
Students with Disabilities - Accommodations and Accessibility
Johns Hopkins University is committed to providing welcoming, equitable, and accessible educational experiences for all students. If disability accommodations are needed for this course, students should request accommodations through Student Disability Services (SDS) as early as possible to provide time for effective communication and arrangements. For further information about this process, please refer to the SDS Website.
Student Conduct Code
The fundamental purpose of the JHU regulation of student conduct is to promote and to protect the health, safety, welfare, property, and rights of all members of the University community as well as to promote the orderly operation of the University and to safeguard its property and facilities. As members of the University community, students accept certain responsibilities which support the educational mission and create an environment in which all students are afforded the same opportunity to succeed academically. For a full description of the code please visit the following website: https://studentaffairs.jhu.edu/policies-guidelines/student-code/
Classroom Climate
JHU is committed to creating a classroom environment that values the diversity of experiences and perspectives that all students bring. Everyone has the right to be treated with dignity and respect. Fostering an inclusive climate is important. Research and experience show that students who interact with peers who are different from themselves learn new things and experience tangible educational outcomes. At no time in this learning process should someone be singled out or treated unequally on the basis of any seen or unseen part of their identity. If you have concerns in this course about harassment, discrimination, or any unequal treatment, or if you seek accommodations or resources, please reach out to the course instructor directly. Reporting will never impact your course grade. You may also share concerns with your program chair, the Assistant Dean for Diversity and Inclusion, or the Office of Institutional Equity. In handling reports, people will protect your privacy as much as possible, but faculty and staff are required to officially report information for some cases (e.g. sexual harassment).
Course Auditing
When a student enrolls in an EP course with “audit” status, the student must reach an understanding with the instructor as to what is required to earn the “audit.” If the student does not meet those expectations, the instructor must notify the EP Registration Team [EP-Registration@exchange.johnshopkins.edu] in order for the student to be retroactively dropped or withdrawn from the course (depending on when the "audit" was requested and in accordance with EP registration deadlines). All lecture content will remain accessible to auditing students, but access to all other course material is left to the discretion of the instructor.