Handout #15 — Pre-training Data

A language model learns from the tokens we actually show it. Choosing those tokens means deciding which sources to collect, which text survives cleaning, and how often repeated content appears. This reading follows a web corpus from crawl to training input, connecting each decision to the distribution learned by the model and the compute spent learning it.

Warm-up: which data reaches the loss?

Predict before opening the answers. These are original, ungraded preparation questions.

1Collecting a corpus: size, coverage, and provenance

1.1 What does “trained on a trillion tokens” count?

Slide 52 illustrates the move from billions to trillions of processed tokens. The useful lesson is the order of magnitude, not a table of model names to memorize. A token count depends on the tokenizer, the text retained, and repetition during training. Two systems reporting the same number of tokens may have seen different amounts of text and very different distributions.

Keep three quantities separate: raw collected text, retained corpus tokens after preparation, and tokens processed during training. With a fixed model and sequence setup, training arithmetic grows roughly with processed tokens. Repeated boilerplate still consumes that budget.

1.2 Why use Common Crawl?

Common Crawl releases large web snapshots collected over time. Reusing a crawl avoids building a crawler, but leaves the hard work of turning web pages into suitable training text. A snapshot is an incomplete sample of the web, shaped by discovery, reachability, and crawl choices. It is not a uniform sample of human language.

Combining snapshots increases coverage, while also adding recrawls, mirrored pages, and older versions. A URL is provenance, not a reliable identity for text: one page may change at the same URL, and the same text may appear at several URLs. Record the snapshot, source URL, extraction version, and later processing decisions so you can explain where retained text came from.

1.3 Read examples before choosing rules

Slide 57 links the instructor’s sample Common Crawl documents. For a short inspection, read several documents and annotate useful content, navigation, repeated templates, extraction failures, language, and domain. A handful of examples suggests hypotheses; it cannot estimate how common a problem is. Test a proposed rule on a separate random sample and on known useful edge cases.

2Cleaning: improve the training signal, then audit the tradeoff

2.1 A heuristic is a testable choice

Slides 59–60 introduce C4’s rule-based cleaning. Examples include requiring terminal punctuation, minimum sentence and word counts, and removing placeholder text such as “Lorem ipsum.” These rules approximate desirable prose; they do not define all useful language. A short answer, poem, equation, or code snippet can be valid even when it fails a prose rule.

Separate extraction (finding the main text), repair (removing a corrupt span), and rejection (dropping a unit). Removing a menu from an article is different from rejecting the whole page because it contains a menu. Likewise, normalization for a matching key need not overwrite the original training text: lowercasing and replacing numbers can erase meaningful distinctions.

2.2 What the C4 comparison establishes

Slide 61 draws on Raffel et al., T5 (2020), §3.4 and Table 8. The unfiltered corpus is larger on disk yet performs worse in the reported comparison. Two columns illustrate the result:

This supports the value of the tested curation recipe in that setup. It does not prove that smaller corpora always win, that every individual rule helps, or that disk bytes measure training tokens. The same study reports task-dependent gains from domain-focused data: a corpus that matches an evaluation domain can help that domain without being the best universal mixture.

2.3 A useful audit has a denominator

For every stage, record input/output documents and tokens, rejection reasons, and affected sources. “We removed 80%” is incomplete: 80% of documents, bytes, or tokens? Relative to the raw crawl or the preceding stage? Sequential stage percentages multiply only when the denominator is clear. For overlapping rules, count both their individual flags and the final unique removals.

To compare training recipes, hold the model, tokenizer, evaluation, and training budget fixed, and report how you matched that budget. A fixed number of steps only matches processed tokens when the effective token batch size also matches. Inspect both aggregate quality and the domains a rule might disproportionately remove.

3Deduplication changes what the model repeatedly learns

3.1 Copies consume compute and alter the objective

Repeated pages, templates, and long copied passages receive repeated next-token losses. With token-averaged training, each presentation contributes to the empirical objective:

Duplicating a document increases the share of terms coming from that document. Deduplication therefore changes both the amount of repetition and the effective weighting of content. In Lee et al. (2022), deduplication reduced memorized output and helped training efficiency; the paper also examines train–validation overlap. Removing overlap makes evaluation more informative, while better held-out loss alone does not establish that all overlap is gone.

3.2 “Duplicate” requires a unit and a matching rule

Whole-document exact matching catches copies but misses a copied paragraph inside otherwise different pages. Paragraph or substring matching finds smaller repeated units, but can also remove legitimate quotations or shared context. Near-duplicate matching catches small edits; it can also merge two versions whose small differences matter. “The dose is 5 mg” and “The dose is 50 mg” must not be treated as interchangeable merely because their words mostly match.

The core choice is semantic and operational: what unit is compared, what counts as equivalent, which population is searched, and what survives? A scalable algorithm implements that policy; it does not decide the right policy for you. §5.2 gives an optional overview of the algorithms on the bonus slides.

3.3 Why removing every repetition can hurt

Repeated boilerplate can crowd out diverse examples, but repeated facts in different useful contexts can reinforce learning. Slide 65’s curve is a conceptual warning, not a measured universal optimum. More removal is not an objective by itself. Tune a policy against quality, coverage, and compute; preserve a record of removed units so you can inspect over-aggressive cases.

If you randomly split a corpus before handling copied content, the same text can land in training and validation. Grouping related copies before splitting helps prevent this particular leakage; verify overlap again afterward, including the granularity you care about. Exact matching alone will not detect paraphrases or every copied fragment.

4Trace the pipeline and interpret its scores

4.1 Each stage changes the input to the next

Slides 87–90 walk through a CCNet-style pipeline: extract text from a crawl, shard it for processing, normalize comparison keys and remove repeated paragraphs, identify language, then score text with a reference LM. Actual recipes differ in their order and retention policies; the diagram is one example, not a mandatory universal pipeline.

A shard is a chunk used to distribute processing. It is not automatically a statistically independent sample. If workers deduplicate only inside their own shards, copies in different shards survive. If you need wider coverage, matching state or an additional comparison stage must cross those boundaries.

4.2 Reference-LM perplexity measures predictability

Recall from Handout #2 that perplexity is the exponential of mean negative log probability. For a reference model $q$ and a scored sequence of length $n$:

A model trained on Wikipedia can score how predictable a passage is under an encyclopedia-like distribution. Extremely high perplexity can flag corrupted text, but also valid code, specialized notation, or another language. Low perplexity can flag familiar or formulaic language; it does not by itself prove that a document is a duplicate. Duplicate detection requires a separate overlap test.

Only compare scores with a consistent model, tokenizer, normalization, and treatment of boundaries. A score of 50 from one tokenizer need not mean the same thing as 50 from another. Averaging paragraph perplexities is also not document perplexity: aggregate log scores and token counts before exponentiating. §6 follows this in real code.

4.3 What should the final corpus report contain?

Keep a stage ledger: source/snapshot versions, extraction and filter settings, before/after token counts, duplicate granularity and scope, and the final sampling policy. Evaluate on held-out data with an overlap audit. Slides 81 and 83 ask two additional questions: how old is the information, and which sources dominate? The detailed methods are optional (§5.3), but recording timestamps and source counts makes these questions answerable.

Static trace: with a minimum of 5 words, the 8-document/45-word input becomes 6 documents/37 words without deduplication; exact matching leaves 5/31; lowercasing and collapsing whitespace leaves 3/19. Documents a, f, and g survive the last setting. The code document has four words and is rejected despite being valid code. The original text is preserved; only the matching key is normalized. The downloadable lab verifies these counts.

5Bonus slides — optional, concise reference

Skip this section on the first pass. It summarizes the blue Bonus slides without requiring algorithm implementation or memorization of dataset recipes. The core practice in §7 does not depend on these details.

6Optional implementation: a tiny audit and real CCNet code

6.1 Run the audit on a CPU

python 15_data_checks.py

The lab checks the widget’s document/word counts, an order-sensitive representative choice, score aggregation, and the small bonus arithmetic. It is an educational miniature: its in-memory set is not a distributed corpus deduplicator, and its word counts are not a model tokenizer.

import re

texts = ["The dose is 5 mg", "The dose is 50 mg"]
def match_key(text):
    return re.sub(r"\d+", "<NUM>", " ".join(text.lower().split()))

for text in texts:
    print(match_key(text))
print("Same key:", match_key(texts[0]) == match_key(texts[1]))

6.2 Code reading: log scores become perplexity

cc_net/perplexity.py · pp · lines 38–39

def pp(log_score, length):
    return 10.0 ** (-log_score / length)

The logarithms here are base 10. Supplying a summed score of −6 across 3 scored tokens gives $10^2=100$. This matches §4.2 after changing log bases; using math.exp on the same base-10 score gives about 7.389, a different value.

cc_net/perplexity.py · DocLM.do · lines 217–220 (inside the loop)

            log_score = model.score(line)
            length = len(line.split()) + 1
            doc_log_score += log_score
            doc_length += length

The code accumulates scores and lengths, including an end-of-sentence position, before computing document perplexity. Its input can already be tokenized pieces; splitting on spaces here does not imply that raw whitespace words are the LM vocabulary. Trace PerplexityBucket.get_bucket next: it labels records rather than deleting them. Find where a later pipeline chooses which labels to retain.

# Optional: check pp from a downloaded copy of the pinned source.
python 15_data_checks.py /path/to/cc_net/perplexity.py

7Core practice and synthesis

Attempt these after §1–§4. Questions are original except the explicitly credited adaptation in K; no answer requires the bonus algorithms or the optional code lab.