CS 601.471/671: Self-supervised Models

Assignments

The homework is your opportunity to practice doing the thing. The lectures and office hours hopefully provide good intuition and motivation and justification for the skills we want you to develop, but the best way to develop those skills is by trying to solve the problems yourself. The practice is far more important than the solution.

The course has ~12 weekly assignments which will improve both your theoretical understanding and your practical skills. All assignments contain both written questions and programming parts (mainly in Python). They will be released on this website, and submissions should be uploaded to Gradescope.

Here is a tentative list of topics for the assignments:

#	Focus
#1	Algebra, calculus, probability recap, implementing Skip-Gram model, classification, evaluation, comparison to basic features (unigrams, bigrams) and existing word embeddings.
#2	Understanding softmax function, classification via vector representations, playing with gradient descent.
#3	PyTorch introduction, automatic differentiation, computation graph, how to use PyTorch on GPUs, basic feedforward network and backpropagation, Word2vec as a feedforward net with automatic differentiation
#4	Neural language model with feedforward network, evaluating language modeling, comparison to count-based model, comparing the representation to Word2vec
#5	Recurrent neural language model and evaluation, comparison to all the earlier models
#6	Transformer-based language model, comparison to the earlier models, distributed training
#7	Decoding language models: greedy, nucleus, typical; various issues related to text generation.
#8	Fine-tuning, prefix-tuning, adapting existing models, comparison to the earlier model; [mis]-Interpreting continuous prompts.
#9	Modifying Transformer for long contexts
#10	In-context learning; dealing with uncertainties
#11	Retrieval-augmented language models
#12	Alignment with human feedback

Submission: Assignments must be submitted via GradeScope. If you're not familiar with Gradescope, check out its documentation.
Collaboration: Study groups are allowed, but students must understand and complete their own assignments, and hand in one assignment per student. If you worked in a group, please put the names of the members of your study group at the top of your assignment. Please ask if you have any questions about the collaboration policy. Again, you must understand and complete your own assignments in your own words, and hand in one assignment per student.
Using Other Resources: We strongly encourage you to use any outside source at your disposal, provided you use your sources properly and give them proper credit. If you get an idea from an outside source, citing that source will not lower your grade. Failing to properly cite an outside source—thereby taking credit for ideas that are not your own—is plagiarism.
Appropriate Citations: You must write everything in your own words, and properly cite every outside source you use, including other students. Using ideas from other sources or people without citation is plagiarism. Copying other sources verbatim, even with proper citation, is plagiarism. Don't do that. The only sources that you are not required to cite are the official course materials (lectures, slides, and assignments).
Honor code: We expect students to not look at solutions or implementations online. We take the student Honor Code seriously. We sometimes use automated methods to detect overly similar assignment solutions.
Late days: Each student has 10 late days to use for assignments. A late day extends the deadline 24 hours. You can use up to 7 late days per assignment (so, if you're late on HW1, you can delay it until the release of HW2). Once you run out of late days but submit late anyway, the submission will not award any points. Once you have used all your late days, the penalty is 5% off the final homework grade for each additional late day. For example, if you're late for 3 days on an assignment (beyond your legitimate "late days" capacity, which is 7 days per homework and 10 days in total), you will lose 15% of the points for that assignment.
Grading: Homeworks are graded by the entire course staff, directly within Gradescope. To keep grading consistent, each numbered problem is graded by a single grader, under the supervision of one of the TAs, using a detailed rubric developed within Gradescope. Under normal circumstances, all homework should be graded within 10 calendar days of submission.
Regrading: Regrade requests can be submitted directly within Gradescope and must include a brief written justification for the request. We encourage students who have questions or concerns about their grades to talk with the course staff before submitting a regrade request. However, no grades will be changed in any student's presence.

Midterm exam

There will be one in-class midterm. The details of this exam is TBD.

Final project

The objective of the final project is to make use of what you have learned during this course to solve a hard problem.

The project deliverables are: (1) A project proposal, (2) a final report, (3) a final project poster summarizing the technical aspects of the project.

Topic: The topic of this project is open-ended. This project, for example, can focus on demonstrating systemic limitations of prior work or suggesting improvements on methods or benchmarks discussed in the class.
Group work: Students are encouraged to work in groups on the final project (team sizes limited to 2 or 3 people).
Project proposals: All groups will be required to submit a project proposal (due on the class calendar). The project proposal is a single-paragraph description of what you intend to do (experiments, datasets, methods, etc.) The instructor(s) will provide feedback on these ideas to help the teams with finding a concrete idea.
Midway progress presentation: Groups will present their halfway progress during a mid-way project presentation (date mentioned on the calendar). This is expected to be a short presentation discussing the progress made for each project and any remaining hurdles.
Final poster: All students will present their findings at a poster presentation during the final exam period.
Final Report: Students should write code and carry out additional experiments and then write up the results in a standard conference paper format (template). The length of the final report should be 5 to 8 pages. References don't count toward the page limit. Note that longer reports are not necessarily better. Students in groups are required to include a “contributions” section concretely lists each author’s contributions (see Section 8 of this paper, for example). The final report should concisely summarize your findings and answer the following questions: 1. What approach did you take to address this problem, and why? 2. How did you explore the space of solutions? 3. How did you evaluate the performance of the approach(es) you investigated? 4. What worked, what did not work, and why?
Project grading: The goal of the project is to demonstrate the group’s understanding of the tools and challenges when using self-supervised models. Grading will reflect the quality of the approach, the rigor of evaluation, and reasoning about successes and failures. Grading will also depend on the completeness of the project, the clarity of the writeup, the level of complexity/difficulty of the approach, and your ability to justify the choices you made. Here is the grade breakdown for the projects:
- Project proposal write-up: 5%
- Quality of final report write-up, implementation and results: 70%
- Final poster and its presentation: 25%

Content Schedule

Each session will involve an instructor-led presentation on a focused topic self-supervised models. There will be weekly assignments related to class presentations, a midterm exam, and a final project.

The current class schedule is below (subject to change):

Date	Topic	Course Materials	Events	Deadlines
#1 - Tue Jan 24	Course overview Plan and expectations [slides]		HW1 released! [tex] [pdf] [colab]
⬇️ -- Self-supervised Word Representations
#2 - Thu Jan 26	Human language and word meaning Word2vec overview Word2vec objective function [slides]	Suggested Reading: Jurafsky & Martin Chapter 6 Additional Reading: Python / Numpy Tutorial (with Jupyter and Colab) Learning Word Embeddings Dive into Deep Learning: Word Embeddings Dive into Deep Learning: Gradient Descent Efficient Estimation of Word Representations in Vector Space (Original W2V paper)
Fri Jan 27	TA Review Session (virtual over Zoom): Math background + Python [zoom link] [slides]	Time: 9 - 9:50 AM
#3 - Tue Jan 31	Word2vec objective function (continued) Inspecting the resulting word vectors Evaluating word vectors [slides]	Suggested Reading: Jurafsky & Martin Chapter 6 Additional Reading: Optimization: Stochastic Gradient Descent Distributed Representations of Words and Phrases and their Compositionality (Negative Sampling paper)	HW2 released! [tex] [pdf] [colab]	HW1 due
⬇️ -- Self-Supervised Representation of Feedforward Neural Language Models
#4 - Thu Feb 2	Word2vec limitations Feedforward networks Neural nets: brief history Word2vec as simple feedforward net [slides]	Suggested Reading: Jurafsky & Martin Chapter 7 Additional Reading: Neural Networks: the Architecture Neural Networks: data and loss Dive into Deep Learning: Multilayer Perceptron Dive into Deep Learning: Practitioners Guide to Neural Networks Dive into Deep Learning: Linear Algebra in PyTorch
Fri Feb 3	TA Review Session (virtual over Zoom): Backpropagation and PyTorch	Time: 9 - 9:50 AM
#5 - Tue Feb 7	Analytical backpropagation Automatic differentiation Practical tips for training neural networks [slides]	Suggested Reading: Jurafsky & Martin Chapter 7 Additional Reading: Neural Networks: Backpropagation Neural Networks: Training and empirical tips Computing Neural Network Gradients Learning representations by back-propagating errors (the original backpropagation paper)		HW2 due
#6 - Thu Feb 9	Extending Word2Vec with FFNs Language modeling objective N-gram language modeling Measuring LM quality Language Modeling with FFN [slides]	Suggested Reading: Jurafsky & Martin Chapter 7 Additional Reading: Jurafsky & Martin Chapter 3 Revisiting Simple Neural Probabilistic Language Models
⬇️ -- Self-Supervised Representation of Recurrent Neural Language Models
#7 - Tue Feb 14	Limitations of feedforward language models Recurrent neural networks Sequence-to-sequence model Training Seq2Seq models Aside: Input units: words, subwords, characters, bytes [slides]	Suggested Reading: CS231N course notes on RNNs Additional Reading: CS224N course notes on RNNs Dive into Deep Learning: Recursive Neural Networks The Unreasonable Effectiveness of Recurrent Neural Networks (blog post overview) Learning long-term dependencies with gradient descent is difficult (one of the original vanishing gradient papers)
#8 - Thu Feb 16	Generation and surprisal Text generation algorithms: - greedy decoding, - stochastic (top-k, nucleus) - exhaustive search - beam search [slides]
⬇️ -- Self-Supervised Representation of Transformer Language Models
#9 - Tue Feb 21	Self-attention: how it works Positional embeddings Computational complexity Self-attention: hacks and variants [slides]	Suggested Reading: Attention Is All You Need Additional Reading: Dive into Deep Learning: Attention Mechanism The Annotated Transformer The Illustrated Transformer
#10 - Thu Feb 23	Transformer architecture Various pre-training objective functions Existing models: BERT, RoBERTa, T5, GPT Scaling laws Multilingual properties Pre-training hacks [slides]	Suggested Reading: TBD Additional Reading: The Illustrated BERT, ELMo, and co Exploring the limits of transfer learning with a unified text-to-text transformer BART: Denoising Sequence-to-Sequence Pre-training
TBD	PyTorch + Backprop Review Session (virtual)	Time: TBD
⬇️ -- Doing Things with Language Models
#11 - Tue Feb 28	Adapting models with parameter change Fine-tuning Prefix-tuning Adaptors [slides]
#12 - Thu Mar 2	Adapting models with prompting In-context learning Multi-step reasoning Connection to supervised learning Failure modes of in-context learning
#13 - Tue Mar 7	Hallucination issue Calibrating model uncertainties - Consistency - Sensitivity - Mutual information - Flatness
⬇️ -- Language Models and Long Context
#14 - Thu Mar 9	Modifying self-attention for long context Retrieval: - Review of retrieval - Dense retrieval - Case study: answering factual questions
#15 - Tue Mar 14	Retrieval augmented language models	Tips on course project
⬇️ -- Social Concerns and Alignment with Human Values
#16 - Thu Mar 16	Bias, fairness and toxic language Moral frameworks and reasoning Truthfulness and veracity	Illustrating Reinforcement Learning from Human Feedback
#17 - Tue Mar 21	No Class - Spring Break
#18 - Thu Mar 23	No Class - Spring Break
#19 - Tue Mar 28	The alignment problem and challenges Alignment as human-in-loop feedback Supervised learning with human judgement Reinforcement learning with human judgements
#20 - Thu Mar 30	Reinforcement learning with human judgements (continued) Evaluating alignment Remaining challenges
⬇️ -- Memorization, Security and Privacy
#21 - Tue Apr 4	Memorization in language models Quantifying memorization Concerns for applications
#22 - Thu Apr 6	Mitigating memorization of private information
⬇️ -- Self-Supervised Learning Text and Other Modalities
#23 - Tue Apr 11	Joint representation of language and programming languages
#24 - Thu Apr 13	Language grounded in structured layouts (web pages, phone apps, ...)
#25 - Tue Apr 18	Joint representation of language and visual information
#26 - Thu Apr 20	Joint representation of text and speech signals
⬇️ -- Future of Self-Supervised Models
#27 - Tue Apr 25	Bias/fairness concerns and feedback loops Environmental concerns Privacy and security issues Legal issues
#28 - Thu Apr 27	Future of self-supervised models Availability of data Availability of compute Limitations and open problems [slides]
#29 - Tue May 2	No Class - Reading Days
#30 - Thu May 4	No Class - Reading Days
TBD	Final project presentation
TBD	Final report submission deadline

Reference text

There is no required text. Though the following can be useful:

Relevant Resources

Here are several resources available for free:

Compute resources:
- Grad students should have access to the graduate grid which has GPUs.
- Undergraduate students should have access to the undergrad grid.
- Google Colab provides free GPU usage for up to 12 hours/day for academic purposes. One can obtain more compute on Colab with relatively minimal pay.
- Google offers research TPU credits.
- Kaggle offers GPUs for its users.
- AWS and Azure both offer welcome credits to students.
- If you need credits to use GPT3, discuss it with the instructor.
Demos:
Tutorials:

PyTorch documentation
These tutorials do a good job of introducing PyTorch.
A course on Huggingface's Transformers library.
Tutorial on Huggingface's Diffusers library
Dive into Deep Learning: Interactive deep learning book with code, math, and discussions.
Tutorials on Learn with Torch Lightning

Besides these resources, we will try our best to satisfy individual needs through discussion.

Conduct

The strength of the university depends on academic and personal integrity. In this course, you must be honest and truthful, abiding by the Computer Science Academic Integrity Policy:

Cheating is wrong. Cheating hurts our community by undermining academic integrity, creating mistrust, and fostering unfair competition. The university will punish cheaters with failure on an assignment, failure in a course, permanent transcript notation, suspension, and/or expulsion. Offenses may be reported to medical, law or other professional or graduate schools when a cheater applies. Violations can include cheating on exams, plagiarism, reuse of assignments without permission, improper use of the Internet and electronic devices, unauthorized collaboration, alteration of graded assignments, forgery and falsification, lying, facilitating academic dishonesty, and unfair competition. Ignorance of these rules is not an excuse.

Academic honesty is required in all work you submit to be graded. Except where the instructor specifies group work, you must solve all homework and programming assignments without the help of others. For example, you must not look at anyone else’s solutions (including program code) to your homework problems. However, you may discuss assignment specifications (not solutions) with others to be sure you understand what is required by the assignment. If your instructor permits using fragments of source code from outside sources, such as your textbook or on-line resources, you must properly cite the source. Not citing it constitutes plagiarism. Similarly, your group projects must list everyone who participated.

In the above paragraph "outside sources" also include content that was produced by an AI assistant like ChatGPT. This follows either by treating the AI assistant as a person for the purposes of this policy (controversial) or acknowledging that the AI assistant was trained directly on people's original work. Thus, while you are not forbidden from using these tools, you should consider the above policy carefully and quote where appropriate. Assignments that are in large part quoted from an AI assistant are very unlikely to be evaluated positively. In addition, if a student's work is substantially identical to another student's work, that will be grounds for an investigation of plagiarism regardless of whether the prose was produced by an AI assistant.

Falsifying program output or results is prohibited. Your instructor is free to override parts of this policy for particular assignments. To protect yourself: (1) Ask the instructor if you are not sure what is permissible. (2) Seek help from the instructor, TA or CAs, as you are always encouraged to do, rather than from other students. (3) Cite any questionable sources of help you may have received.

Report any violations you witness to the instructor. You can find more information about university misconduct policies on the web for undergraduates and undergraduates students.

Johns Hopkins University is committed to equal opportunity for its faculty, staff, and students. To that end, the university does not discriminate on the basis of sex, gender, marital status, pregnancy, race, color, ethnicity, national origin, age, disability, religion, sexual orientation, gender identity or expression, veteran status, military status, immigration status or other legally protected characteristic. The University's Discrimination and Harassment Policy and Procedures provides information on how to report or file a complaint of discrimination or harassment based on any of the protected statuses listed in the earlier sentence, and the University’s prompt and equitable response to such complaints.

CS 601.471/671 NLP: Self-supervised Models

Johns Hopkins University - Spring 2023

Logistics

Key links