CS 601.471/671: Self-supervised Models

Large self-supervised (pre-trained) models (such as LLMs) have transformed various data-driven fields, such as natural language processing (NLP). Its objective is to provide a comprehensive overview of the industry standards for training and deploying language models. The course is designed to equip you with a diverse set of skills essential for future success either in academia or industry. PyTorch will be the primary programming language used in the class. The course will include instructor-led lectures, multiple in-class quizzes, and a final project.

Note: The course is different from 601.771 (typically offered in fall semesters) which is focused on advanced topics in recent papers and is geared toward grad students that want to specialize in the latest developments.

Prerequisites:

Background in algorithms: "Data Structures" (601.226)
All the class assignments will be in Python/PyTorch. If you don’t know Python or PyTorch but have experience in other programming languages (Java, C++, etc.) you can probably pick Python/PyTorch pretty quickly.
Calculus and linear algebra: you should be comfortable with matrix operations (matrix multiplication, transpose, inverse, dot product, gradients).
Probability: basic probability properties (conditionals, marginals, mean, standard deviation), distributions (normal, categorical, etc.).

Suggestive prerequisites: Not a hard prerequisites but it is highly useful to have background in "Natural Language Processing" and "Machine Learning". You may gain such background by taking the following courses: "Machine Learning" (CS 475/675), OR "Machine Learning: Deep Learning" (CS 482/682), OR "Natural Language Processing" (CS 465/665), OR "Machine Translation" (CS 468/668).

Relevant Courses at Hopkins: This course has some overlap with "Natural Language Processing" (EN.601/665), and "Artificial Agents" (EN.601.470/670), though the courses have different focuses.

Logistics

Classes: on Tuesday/Thursday @ 9-10:15 am EST, Hackerman B17
Office hours:
- Daniel office hour: Tuesdays 10:15 am - 11 pm EST / Location: TBD,
- TA office hour: Monday 10:00 AM - 11:00 AM / Location: see below,
- CA office hour: Fridays 10:00 AM - 11:00 AM / Location: see below.
The TA/CA office hours will be held at Hackerman 3rd Floor common area which is the whiteboard area between Hackerman 323 and 324 (to be confirmed). The TA/CAs will also be available on Zoom during the office hours.
Contact: If you have any questions about the course, you can post them on Piazza. Please avoid message CAs/TA directly unless necessary!
Changes: The instructor reserves the right to make changes to the syllabus or project due dates. These changes will be announced as early as possible.
News and announcements: All the news and announcements will be made on Piazza.
Course grade: Your grade is based on the following activities: (1) regular in-class quizzes (70%) -- done individually, closed-book and in-class; (2) a final project (30%) -- done in groups and same grade for all members of a team. Attendance is *not* mandatory (hence, 0%) but highly encouraged: participation in class is our chance to learn more effectively.

Key links

Piazza for discussion and announcements. Sign up, follow, ask questions, and participate in discussions!
Gradescope for submitting your assignments.

Assignments

We don't have any homework assignments that we would grade. You should, however, read and undersand the lectures and the assigned reading ("Additional Reading" column in the schedule below).

In-class Quizzes

What are the in-class quizzes? There will be multiple in-class quizzes. These will be paper-based and during the usual class time.
What is the purpose of the quizzes? They assess students' mastery of the topics discussed in the lectures and assigned readings. They also provide feedback to both the student and the instructor, and identify areas that need improvement to inform further learning and teaching.
What material do the quizzes cover? The quizzes will encompass all material from the start of the semester up to the topic specified by the instructor.
How long are the quizzes? All quizzes (except the final one) will be short (2-3 questions).
What happens if I skip a quiz? You may skip up to 2 quizzes without notifying the instructor. We will distribute the weight of these skipped quizzes to the remaining quizzes you take. If you skip more than 2 quizzes, you lose their grade.
Where can I find examples of past quizzes/midterms? You can find them here.

Class Project

The objective of the final project is to make use of what you have learned during this course to solve a hard problem.

The final project milestones include: (1) A project proposal, (2) A project midway report, (3) a final report, (4) a final project poster summarizing the technical aspects of the project. See the course calendar for the due dates.

Topic: See this document.
Group work: Students are strongly encouraged to work in groups on the final project (team sizes limited to a maximum of 4 people).
Project proposals: All groups will be required to submit a project proposal (due on the class calendar). The project proposal is a 2-page description of what you intend to do (experiments, datasets, methods, etc.) All documents (proposal, midway report and final report) should use this template. The instructor(s) will provide feedback on these ideas to help the teams with finding a concrete idea. Here is examples of project proposals from previous years:
- Adversarial Prompting of Unlearned Language Models
- Ensemble Domain-Specific Knowledge Distillation
Midway progress reports: Reports discussing the progress made thus far (at most 5 pages; use the same suggested template for the proposals) and elaborates on the remaining work. Describe the progress made, experiments you have run, preliminary results you have obtained, how you plan to spend the rest of your time, etc. While this is called "midway" in practice it should be considered more than halfway! By this milestone, you’re expected to have implemented something, and to have some experimental results to show by this date.
Final poster presentations: All students will present their findings at a poster presentation during the final exam period.
Final report: Students should write code and carry out additional experiments and then write up the results in a standard conference paper format (at most 8 pages; use the same template we had suggested for the proposals). References don't count toward the page limit. Note that longer reports are not necessarily better. Students in groups are required to include a “contributions” section concretely lists each author’s contributions (see Section 8 of this paper, for example). The final report should concisely summarize your findings and answer the following questions: 1. What approach did you take to address this problem, and why? 2. How did you explore the space of solutions? 3. How did you evaluate the performance of the approach(es) you investigated? 4. What worked, what did not work, and why?
Here are examples of final reports from the previous year:
Project grading: The goal of the project is to demonstrate the group’s understanding of the tools and challenges when using self-supervised models. Grading will reflect the quality of the approach, the rigor of evaluation, and reasoning about successes and failures. Grading will also depend on the completeness of the project, the clarity of the writeup, the level of complexity/difficulty of the approach, and your ability to justify the choices you made. Here is the grade breakdown for the projects:
- Project proposal: 20%
- Midway report: 20%
- Quality of final report write-up, implementation and results: 40%
- Final poster and its presentation: 20%
Can I skip the final project presentation? Yes! But you'd lose the grade final presentation. The final project presentation is essentially a replacement for the final exam and is in-person. The date is determined by the registrar (out of my control). You have the option to skip it in which case you (not your team) would lose the grade for the presentation (20% of the total project grade) which is 4% (0.2x0.2) of the total grade.

Content Schedule

The current class schedule is below (subject to change):

Date	Topic	Content	Additional Reading	Events
#1 - Tue Sept 1	Course introduction: Course overview Plan and expectations	PPTX, PDF	TBD TBD
#2 - Thu Sept 3	Language modeling: Definitions and history, Counting and n-grams, Measuring LM quality, Language modeling as a learning problem	PPTX, PDF	TBD TBD
#3 - Tue Sept 8	Neural Nets: Definitions Brief history Background (algebra + optimization)	PPTX, PDF	TBD TBD	📝 QUIZ #1 - Topics: everything discussed up to this point.
#4 - Thu Sept 10	Neural Nets: Background (algebra + optimization) Analytical backprop	PPTX, PDF	TBD TBD
#5 - Tue Sept 15	Neural Nets: Analytical backprop Backprop in practice	PPTX, PDF	TBD TBD
#6 - Thu Sept 17	Neural Nets: Backprop in practice Practical tips	PPTX, PDF	TBD TBD	📝 QUIZ #2 - Topics: everything discussed up to this point.
#7 - Tue Sept 22	Simple neural language models: Tokenization and subwords Fixed-window MLP LMs	PPTX, PDF	TBD TBD
#8 - Thu Sept 24	Recurrent Neural LMs: Introducing RNNs Training RNNs RNNs: Pros and Cons Bonus: Pre-training RNNs	PPTX, PDF	TBD TBD
#9 - Tue Sept 29	Sampling from LMs	PPTX, PDF	TBD TBD	📝 QUIZ #3 - Topics: everything discussed up to this point.
#10 - Thu Oct 1	Transformer architecture: Self-attention	PPTX, PDF	TBD TBD
#11 - Tue Oct 6	Transformer architecture: Self-attention A decoder-only Transformer	PPTX, PDF	TBD TBD
#12 - Thu Oct 8	Transformer architecture: Transformer architectural variants Computational cost	PPTX, PDF	TBD TBD	📝 QUIZ #4 - Topics: everything discussed up to this point.
#13 - Tue Oct 13	Transformer-based language models: Notable models Architectural variants	PPTX, PDF	TBD TBD
#14 - Thu Oct 15	Transformer-based language models: Architectural variants LayerNorm variants FFN variations: activations, bias and activations KV-drag and Self-attention variants	PPTX, PDF	TBD TBD
#15 - Tue Oct 20	Transformer-based language models: Architectural variants Parameter tying Positional encoding Dimensions Tokenizers Pre-training data	PPTX, PDF	TBD TBD	📝 QUIZ #5 - Topics: everything discussed up to this point.
Thu Oct 22	No Class - Fall Break
#16 - Tue Oct 27	Transformer-based language models: Pre-training data Pre-training optimization Adapting LMs: Adaption as fine-tuning Parameter-efficient tuning	PPTX, PDF	TBD TBD
#17 - Thu Oct 29	Adapting LMs: Adaption as fine-tuning Parameter-efficient tuning Adaption as in-context learning Prompt engineering Multi-step prompting Failures of prompting	PPTX, PDF	TBD TBD
#18 - Tue Nov 3	tbd			📝 QUIZ #6 - Topics: everything discussed up to this point.
#19 - Thu Nov 5	Adaption as in-context learning Prompt engineering Multi-step prompting Failures of prompting Introducing final projects: Defining final projects Tips for successful project	PPTX, PDF PPTX, PDF	TBD TBD
#20 - Tue Nov 10	Alignment of LMs: Alignment: definitions Instruction-tuning	PPTX, PDF	TBD TBD
Apr 4	Project proposals deadline
#21 - Thu Nov 12	Alignment of LMs: RLHF as vanilla policy gradient	PPTX, PDF	TBD TBD	📝 QUIZ #7 - Topics: everything discussed up to this point.
#22 - Tue Nov 17	Alignment of LMs: Alignment: failures/open questions RLHF variants Inference-scaling: The reward function The inference algorithms Training a good inference-scaler	PPTX, PDF PPTX, PDF	TBD TBD
#23 - Thu Nov 19	Efficiency considerations: Distributed training	PPTX, PDF	TBD TBD
Tue Nov 24	No Class - Fall Recess
Thu Nov 26	No Class - Fall Recess / University Holiday
#24 - Tue Dec 1	Efficiency considerations: Distributed training	PPTX, PDF	TBD TBD	📝 QUIZ #8 - Topics: everything discussed up to this point.
#25 - Thu Dec 3	Efficiency considerations: Distributed training Quantization Distillation	PPTX, PDF	TBD TBD
#26 - Tue Dec 8	tbd
#27 - Thu Dec 10	Final quiz			📝 QUIZ #9 - Final quiz — last day of class.
TBD	Final project reports
Thu May 8	Final project poster session (Time: 2pm-5pm)

Relevant Resources

Here are several resources available for free:

Compute resources:
- Google Colab provides free GPU usage for up to 12 hours/day for academic purposes. One can obtain more compute on Colab with relatively minimal pay.
- Google offers research TPU credits.
- AWS and Azure both offer welcome credits to students.
- If you need credits to use GPT3/GPT4 or other APIs, discuss it with the instructor.
Tutorials:

A course on Huggingface's Transformers library.
Tutorials on Learn with Torch Lightning

Besides these resources, we will try our best to satisfy individual needs through discussion.

Code of Conduct

The strength of the university depends on academic and personal integrity. In this course, you must be honest and truthful, abiding by the Computer Science Academic Integrity Policy:

Cheating is wrong. Cheating hurts our community by undermining academic integrity, creating mistrust, and fostering unfair competition. The university will punish cheaters with failure on an assignment, failure in a course, permanent transcript notation, suspension, and/or expulsion. Offenses may be reported to medical, law or other professional or graduate schools when a cheater applies. Violations can include cheating on exams, plagiarism, reuse of assignments without permission, improper use of the Internet and electronic devices, unauthorized collaboration, alteration of graded assignments, forgery and falsification, lying, facilitating academic dishonesty, and unfair competition. Ignorance of these rules is not an excuse.

Academic honesty is required in all work you submit to be graded. Except where the instructor specifies group work, you must solve all homework and programming assignments without the help of others. For example, you must not look at anyone else’s solutions (including program code) to your homework problems. However, you may discuss assignment specifications (not solutions) with others to be sure you understand what is required by the assignment. If your instructor permits using fragments of source code from outside sources, such as your textbook or on-line resources, you must properly cite the source. Not citing it constitutes plagiarism. Similarly, your group projects must list everyone who participated.

In the above paragraph "outside sources" also include content that was produced by an AI assistant like ChatGPT. This follows either by treating the AI assistant as a person for the purposes of this policy (controversial) or acknowledging that the AI assistant was trained directly on people's original work. Thus, while you are not forbidden from using these tools, you should consider the above policy carefully and quote where appropriate. Assignments that are in large part quoted from an AI assistant are very unlikely to be evaluated positively. In addition, if a student's work is substantially identical to another student's work, that will be grounds for an investigation of plagiarism regardless of whether the prose was produced by an AI assistant.

Falsifying program output or results is prohibited. Your instructor is free to override parts of this policy for particular assignments. To protect yourself: (1) Ask the instructor if you are not sure what is permissible. (2) Seek help from the instructor, TA or CAs, as you are always encouraged to do, rather than from other students. (3) Cite any questionable sources of help you may have received.

Report any violations you witness to the instructor. You can find more information about university misconduct policies on the web for undergraduates and graduates students.

Johns Hopkins University is committed to equal opportunity for its faculty, staff, and students. To that end, the university does not discriminate on the basis of sex, gender, marital status, pregnancy, race, color, ethnicity, national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, military status, immigration status or other legally protected characteristic. The University's Discrimination and Harassment Policy and Procedures provides information on how to report or file a complaint of discrimination or harassment based on any of the protected statuses listed in the earlier sentence, and the University’s prompt and equitable response to such complaints.

CS 601.471/671 NLP: Self-supervised Models

Johns Hopkins University - Fall 2026