CS 601.771: Advances in Self-supervised Models

Large self-supervised (pre-trained) models (such as LLMs) have transformed various data-driven fields, such as natural language processing (NLP). This advanced course aims to offer a more in-depth knowledge of these technologies.

Specifically, this year we will focus on the following topics:

Reasoning: Understanding LLMs ability and limitations in performing complex, multi-step inference.
Long inputs/output: Understanding LLMs ability and limitations in processing long inputs and generating long outputs.
Efficient inference: Understanding the latest advances and hurdles in efficient inference.

Note: The course is different from (more advanced than) 601.471/671 (offered in the spring semesters) which is focused on building the foundational concepts.

Prerequisites: Natural Language Processing (CS 465/665), NLP: Self-Supervised Models (CS 471/671), or instructor consent.

Relevant Courses at Hopkins: This course has some overlap with "Natural Language Processing" (EN.601/665), and "Artificial Agents" (EN.601.470/670), though the courses have different focuses.

Logistics

Classes: on Tuesday/Thursday 4:30 - 5:45 pm EST (room: Malone 228)
Office hours: Daniel office hour: by appointment (Hackerman hall, 316B).
Contact: If you have any questions about the course, you can post them on Slack.
Virtual or in-person: The class will be in-person.
News and announcements: All the news and announcements will be made on Slack.

Key links

Slack for discussion and announcements. Sign up, follow, ask questions, and participate in discussions!
Gradescope for submitting your reports (entry code: PG87D2).

Format Overview

There will be student-driven discussion and critique sessions in which we go over and discuss selected papers in each area. For each paper, ~two students will be assigned to write a review describing the ‘pros’ and ‘cons’ of the paper, and they will present their critique during the discussion.

Evaluation and Format Details

One assignment (10%) -- individually: The course has ONE assignment to measure your understanding of the foundational concepts of self-supervised learning. This is to make sure that when coming in, you know all the pre-requisites needed for the class. They will be released on this website, and submissions should be uploaded to Gradescope.
- Submission:The assignment must be submitted through Gradescope.
- Regrading: Regrade requests can be submitted directly within Gradescope and must include a brief written justification for the request.
- Late days: There are no late days for the one homework assignment.
In-class participation (20%) -- individually. This includes engagement and punctuality (being present in class before the presentation starts). It is important that you attend each session, complete the readings prior to class to participate in the discussions during the class.
- Step Up/Step Back: This means, if you are the person who feels very comfortable sharing, take note of how often you are sharing, and consider giving time for others to share (make sure to not monopolize the class time). By all means, be present and active in this conversation, but make sure others have the time to as well. If you tend to be a quiet participant, take a chance and “step up” with your idea, share your concerns, your ideas, concerns, and excitement with the group. A good facilitator will make sure this is safe for you.
- Punctuality: We expect you to arrive before the presentations start. Late arrivals will negatively impact your participation grade.
One-page summary and presentation (20%) -- individually. One-page summaries are meant to be mechanism to force us to think about what learned during the class.
- After the class: After each class, each student will write a "one-pager" summary of what this class was about and what they learned (due the night before the class). They should organize it so that one can quicky see the main takeaways.
- Beginning of the class: At the beginning of each class, one of the students (randomly selected) will remind us what we talked about in the previous class based on their one-pager.
Student-led paper presentation and critique (20%) -- in group. This includes presenting papers assigned to you. Most class will involve a ~30-minute presentation by a team of 2-3 students about 1-2 assigned papers. The teams/paper assignments will be communicated at least 10 days ahead of the presentation.
- Slide feedback: The presenting team must share their slides ahead of time with the course staff 48 hours before the presentation (Slack channel: #presentation-feedback). We will review the slides and provide feedback about its content.
- Late days: There are no late days for paper presentations. You cannot miss your presentation day. If you can't make it, you're responsible for communicating it well in advance (>2 weeks earlier).
The course project (30%) -- teams or individually. See the paragraph below for details
Skip days: You may skip up to 3 classes (during the dates that you're not presenting) with instructor consent ahead of time.

Course Project

The objective of the final project is to make use of what you have learned during this course to solve a hard problem.

The final project milestones include: (1) A project proposal, (2) A project midway report, (3) a final report, (4) a final project poster summarizing the technical aspects of the project. See the course calendar for the due dates.

Topic: The topic of this project is open-ended, as long as it's related to the course's focus. For example, the project may involve one method or a setting covered in the course.
Group work: Students are strongly encouraged to work in groups on the final project (team sizes limited to a maximum of 4 people).
Project proposals: All groups will be required to submit a project proposal (due on the class calendar). The project proposal is a 2-page description of what you intend to do (experiments, datasets, methods, etc.) All documents (proposal, midway report and final report) should use this template. The instructor(s) will provide feedback on these ideas to help the teams with finding a concrete idea. Here is examples of project proposals from previous years:
- Adversarial Prompting of Unlearned Language Models
- Ensemble Domain-Specific Knowledge Distillation
Midway progress reports: Reports discussing the progress made thus far (at most 5 pages; use the same suggested template for the proposals) and elaborates on the remaining work. Describe the progress made, experiments you have run, preliminary results you have obtained, how you plan to spend the rest of your time, etc. While this is called "midway" in practice it should be considered more than halfway! By this milestone, you’re expected to have implemented something, and to have some experimental results to show by this date.
Final poster presentations: All students will present their findings at a poster presentation during the final exam period.
Final report: Students should write code and carry out additional experiments and then write up the results in a standard conference paper format (at most 8 pages; use the same template we had suggested for the proposals). References don't count toward the page limit. Note that longer reports are not necessarily better. Students in groups are required to include a “contributions” section concretely lists each author’s contributions (see Section 8 of this paper, for example). The final report should concisely summarize your findings and answer the following questions: 1. What approach did you take to address this problem, and why? 2. How did you explore the space of solutions? 3. How did you evaluate the performance of the approach(es) you investigated? 4. What worked, what did not work, and why?
Here are examples of final reports from the previous year:
Project grading: The goal of the project is to demonstrate the group’s understanding of the tools and challenges when using self-supervised models. Grading will reflect the quality of the approach, the rigor of evaluation, and reasoning about successes and failures. Grading will also depend on the completeness of the project, the clarity of the writeup, the level of complexity/difficulty of the approach, and your ability to justify the choices you made. Here is the grade breakdown for the projects:
- Project proposal: 20%
- Midway report: 20%
- Quality of final report write-up, implementation and results: 40%
- Final poster and its presentation: 20%
Using Other Resources: We strongly encourage you to use any outside source at your disposal, provided you use your sources properly and give them proper credit. If you get an idea from an outside source, citing that source will not lower your grade. Failing to properly cite an outside source—thereby taking credit for ideas that are not your own—is plagiarism.
AI can be used? Of course! Chatbots have become essential ingredients of daily workflows. You're welcome to use in various stages of your project to augment your productivity.
Appropriate Citations: You must write everything in your own words, and properly cite every outside source you use, including other students. Using ideas from other sources or people without citation is plagiarism. Copying other sources verbatim, even with proper citation, is plagiarism. Don't do that. The only sources that you are not required to cite are the official course materials (lectures, slides, and assignments).
Can it become a paper? It often takes a lot of effort to turn a class project into a publishable work. But it's certainly feasible (prior students have done it; e.g., this or this or this). In terms of timing, it's something that can be discussed after the semester is over.
Can I skip the final project presentation? Sure! But you'd lose the grade final presentation. The final project presentation is essentially a replacement for the final exam and is in-person. The date is determined by the registrar (out of our control). You have the option to skip it in which case you (not your team) would lose the grade for the presentation.

Content Schedule

The current class schedule is below (subject to change).

Date	Research questions	Papers and/or slides
#1 - Tue Aug 26	Reviewing the foundation: Course overview Plan and expectations	[slides: pptx, pdf]
Aug 26	HW1 is released!
#2 - Thu Aug 28	Reviewing the foundation: Evaluation Transformers	[slides: pptx, pdf] [slides: pptx, pdf]
#3 - Tue Sept 2	Reviewing the foundation: Pre-training
Sept 4	HW1 deadline
#4 - Thu Sept 4	Reviewing the foundation: TBD
#5 - Tue Sept 9	Reviewing the foundation: TBD
#6 - Thu Sept 11	TBD
#8 - Tue Sept 16	TBD
#9 - Thu Sept 18	TBd
#9 - Tue Sept 23	TBD
Sept 25	Project proposals deadline
#10 - Thu Sept 25	TBD
#11 - Tue Sept 30	TBD
#12 - Thu Oct 2	TBD
#13 - Tue Oct 7	TBD
#14 - Thu Oct 9	TBD
#15 - Tue Oct 14	TBD
Oct 15	Progress report #1 deadline
#16 - Thu Oct 17	No Class - Fall break
#17 - Tue Oct 21	Guest speaker
#18 - Thu Oct 23	Guest speaker
#19 - Tue Oct 28	TBD
#20 - Thu Oct 30	TBD
#21 - Tue Nov 4	TBD
#22 - Thu Nov 6	TBD
Nov 7	Progress report #2 deadline
#23 - Tue Nov 11	TBD
#24 - Thu Nov 13	TBD
#25 - Tue Nov 18	TBD
#26 - Thu Nov 20	TBD
#27 - Tue Nov 25	No Class - Fall Recess
#28 - Thu Nov 27	No Class - Fall Recess
#29 - Tue Dec 2	TBD
#30 - Thu Dec 4	TBD
Dec 8-11	Reading Days
Dec TBD	Final project reports
Dec TBD	Final project poster session (6-9pm) -- final exam schedule

Code of Conduct

The strength of the university depends on academic and personal integrity. In this course, you must be honest and truthful, abiding by the Computer Science Academic Integrity Policy:

Cheating is wrong. Cheating hurts our community by undermining academic integrity, creating mistrust, and fostering unfair competition. The university will punish cheaters with failure on an assignment, failure in a course, permanent transcript notation, suspension, and/or expulsion. Offenses may be reported to medical, law or other professional or graduate schools when a cheater applies. Violations can include cheating on exams, plagiarism, reuse of assignments without permission, improper use of the Internet and electronic devices, unauthorized collaboration, alteration of graded assignments, forgery and falsification, lying, facilitating academic dishonesty, and unfair competition. Ignorance of these rules is not an excuse.

Academic honesty is required in all work you submit to be graded. Except where the instructor specifies group work, you must solve all homework and programming assignments without the help of others. For example, you must not look at anyone else’s solutions (including program code) to your homework problems. However, you may discuss assignment specifications (not solutions) with others to be sure you understand what is required by the assignment. If your instructor permits using fragments of source code from outside sources, such as your textbook or on-line resources, you must properly cite the source. Not citing it constitutes plagiarism. Similarly, your group projects must list everyone who participated.

In the above paragraph "outside sources" also include content that was produced by an AI assistant like ChatGPT. This follows either by treating the AI assistant as a person for the purposes of this policy (controversial) or acknowledging that the AI assistant was trained directly on people's original work. Thus, while you are not forbidden from using these tools, you should consider the above policy carefully and quote where appropriate. Assignments that are in large part quoted from an AI assistant are very unlikely to be evaluated positively. In addition, if a student's work is substantially identical to another student's work, that will be grounds for an investigation of plagiarism regardless of whether the prose was produced by an AI assistant.

Falsifying program output or results is prohibited. Your instructor is free to override parts of this policy for particular assignments. To protect yourself: (1) Ask the instructor if you are not sure what is permissible. (2) Seek help from the instructor, TA or CAs, as you are always encouraged to do, rather than from other students. (3) Cite any questionable sources of help you may have received.

Report any violations you witness to the instructor. You can find more information about university misconduct policies on the web for undergraduates and graduates students.

Johns Hopkins University is committed to equal opportunity for its faculty, staff, and students. To that end, the university does not discriminate on the basis of sex, gender, marital status, pregnancy, race, color, ethnicity, national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, military status, immigration status or other legally protected characteristic. The University's Discrimination and Harassment Policy and Procedures provides information on how to report or file a complaint of discrimination or harassment based on any of the protected statuses listed in the earlier sentence, and the University’s prompt and equitable response to such complaints.

CS 601.771 Advances in Self-supervised Models

Johns Hopkins University - Fall 2025