CS 601.471/671 NLP: Self-supervised Models

Assignments

The homework offers you a chance to practice applying what you've learned. It is intended to complement lectures and office hours, which aim to provide intuition, motivation, and justification for the skills we want you to develop. However, the most effective way to build these skills is by attempting to solve the problems on your own. The process of practicing is far more valuable than simply being provided with the solution.

The course includes 7 weekly assignments leading up to spring break, designed to enhance both your theoretical knowledge and practical abilities. Each assignment consists of written questions and programming components (Python). Assignments will be posted on this website, and submissions should be uploaded to Gradescope.

Here is a tentative list of topics for the assignments:

• Submission: Assignments must be submitted via GradeScope. If you're not familiar with Gradescope, check out its documentation.
• Collaboration: Study groups are permitted, but each student must individually understand and complete their own assignments in your own words (not sharing answers), submitting one assignment per person. Remember, you must comprehend and complete your assignments independently, using your own words, and submit one assignment per student.
• Using other Resources: We highly recommend utilizing any external resources available to you, as long as you properly acknowledge and credit these sources. Citing an external source when you derive an idea from it will not negatively impact your grade. However, neglecting to properly cite an external source and thereby claiming credit for ideas that are not your own constitutes plagiarism.
• Appropriate citations: You must write everything in your own words, and properly cite every outside source you use, including other students. Using ideas from other sources or people without citation is plagiarism. Copying other sources verbatim, even with proper citation, is plagiarism. Don't do that. The only sources that you are not required to cite are the official course materials (lectures, slides, and assignments).
• Honor code: We expect students to not look at solutions or implementations online. We take the student Honor Code seriously. We sometimes use automated methods to detect overly similar assignment solutions.
• Late days: Got sick? Have a conference to attend? Use your "late days"! Each student has 7 late days to use for assignments. A late day extends the deadline 24 hours. Once you have used all your late days, the penalty is 5% off the final homework grade for each additional late day. For example, if you're late for 3 days on an assignment (beyond your legitimate "late days" capacity, which is 7 days per homework and 10 days in total), you will lose 15% of the points for that assignment. The deadline cutoffs are at 9am each day. There are no fractional late days. If you're late for 1 hour, you lose a full day.
• Max delay per assignment: You can use up to 7 late days per assignment (so, if you're late on HW1, you can submit it until the release of HW2). Assignments submitted after 7 late days will not be graded (unless explicit permission is given in advance by the instructor).
• Grading: Homeworks are graded by the entire course staff directly on Gradescope. To keep grading consistent, each numbered problem is graded by a single grader, under the supervision of one of the TAs, using a detailed rubric on Gradescope. Under normal circumstances, all homework should be graded within 10 calendar days of submission.
• Regrading: Regrade requests can be submitted directly within Gradescope and must include a brief written justification for the request. We encourage students who have questions or concerns about their grades to talk with the course staff before submitting a regrade request. However, no grades will be changed in any student's presence.

In-class Quizzes

There will be 3 in-class quizes. The midterm exams will be paper-based and during the usual class time. The exams will assess students' mastery of the topics discussed in the lectures and homework assignments. The exams also provide feedback to both the student and the instructor, and identify areas that need improvement to inform further learning and teaching. The quizzes will encompass all material from the start of the semester up to the topic specified by the instructor.

Class Project

The objective of the final project is to make use of what you have learned during this course to solve a hard problem.

The final project milestones include: (1) A project proposal, (2) A project midway report, (3) a final report, (4) a final project poster summarizing the technical aspects of the project. See the course calendar for the due dates.

• Topic: The topic of this project is open-ended but it must connect either to self-supervised learning (pre-training). This project may involve, for instance, replicating a significant previous study, highlighting systemic limitations of earlier work, or proposing enhancements to methods or benchmarks covered in the class.
• Group work: Students are strongly encouraged to work in groups on the final project (team sizes limited to a maximum of 4 people).
• Project proposals: All groups will be required to submit a project proposal (due on the class calendar). The project proposal is a 2-page description of what you intend to do (experiments, datasets, methods, etc.) All documents (proposal, midway report and final report) should use this template. The instructor(s) will provide feedback on these ideas to help the teams with finding a concrete idea. Here is examples of project proposals from previous years:
  • Adversarial Prompting of Unlearned Language Models
  • Ensemble Domain-Specific Knowledge Distillation
• Midway progress reports: Reports discussing the progress made thus far (at most 5 pages; use the same suggested template for the proposals) and elaborates on the remaining work. Describe the progress made, experiments you have run, preliminary results you have obtained, how you plan to spend the rest of your time, etc. While this is called "midway" in practice it should be considered more than halfway! By this milestone, you’re expected to have implemented something, and to have some experimental results to show by this date.
• Final poster presentations: All students will present their findings at a poster presentation during the final exam period.
• Final report: Students should write code and carry out additional experiments and then write up the results in a standard conference paper format (at most 8 pages; use the same template we had suggested for the proposals). References don't count toward the page limit. Note that longer reports are not necessarily better. Students in groups are required to include a “contributions” section concretely lists each author’s contributions (see Section 8 of this paper, for example). The final report should concisely summarize your findings and answer the following questions: 1. What approach did you take to address this problem, and why? 2. How did you explore the space of solutions? 3. How did you evaluate the performance of the approach(es) you investigated? 4. What worked, what did not work, and why?
  Here are examples of final reports from the previous year:
  • Adversarial Prompting of Unlearned Language Models
  • Ensemble Domain-Specific Knowledge Distillation
  • Efficient Distillation of Transformers via Self-Teaching
• Project grading: The goal of the project is to demonstrate the group’s understanding of the tools and challenges when using self-supervised models. Grading will reflect the quality of the approach, the rigor of evaluation, and reasoning about successes and failures. Grading will also depend on the completeness of the project, the clarity of the writeup, the level of complexity/difficulty of the approach, and your ability to justify the choices you made. Here is the grade breakdown for the projects:
  • Project proposal: 20%
  • Midway report: 20%
  • Quality of final report write-up, implementation and results: 40%
  • Compelling introduction/motivation and clear problem statement and desired outcome: 5%
  • Clear description of methods and evaluation protocol: 5%
  • Clear and complete coverage of related work: 5%
  • The rigor of evaluation and reasoning / discussion: 5%
  • Clear articulation of the results (includes figures, tables): 10%
  • Innovativeness: 5%
  • Discussion and conclusion comprised of well-formulated arguments, grounded in the experimental results and the broader scientific literature: 5%
  • Final poster and its presentation: 20%