Skip to the notes
← All winter school notes

Marianna Nezhurina

Pitfalls in Measuring Generalization

Download PDF ↓

HPLT × NLPL Winter School · Day 2 · 11 February 2025 · Scroll to read

About these notes

Lecture material on evaluation leakage, benchmark contamination, contamination-resistant tests, and the methodological traps that can make language-model generalization look stronger than it is.

  1. Page 1 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 1 of 21
  2. Page 2 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 2 of 21
  3. Page 3 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 3 of 21
  4. Page 4 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 4 of 21
  5. Page 5 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 5 of 21
  6. Page 6 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 6 of 21
  7. Page 7 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 7 of 21
  8. Page 8 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 8 of 21
  9. Page 9 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 9 of 21
  10. Page 10 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 10 of 21
  11. Page 11 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 11 of 21
  12. Page 12 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 12 of 21
  13. Page 13 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 13 of 21
  14. Page 14 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 14 of 21
  15. Page 15 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 15 of 21
  16. Page 16 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 16 of 21
  17. Page 17 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 17 of 21
  18. Page 18 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 18 of 21
  19. Page 19 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 19 of 21
  20. Page 20 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 20 of 21
  21. Page 21 of 21 from Pitfalls in Measuring Generalization, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 21 of 21