Open Foundation Models: Scaling Laws & Generalization

HPLT × NLPL Winter School · Day 2 · 11 February 2025 · Scroll to read

About these notes

Lecture material on open foundation models, empirical scaling laws, compute and data trade-offs, and the limits of using smooth power-law curves to predict real-world generalization.

Foundation models and transferable learning
Scaling with data, parameters, and compute
Generalization limits and open-model research

Page 1 of 100
Page 2 of 100
Page 3 of 100
Page 4 of 100
Page 5 of 100
Page 6 of 100
Page 7 of 100
Page 8 of 100
Page 9 of 100
Page 10 of 100
Page 11 of 100
Page 12 of 100
Page 13 of 100
Page 14 of 100
Page 15 of 100
Page 16 of 100
Page 17 of 100
Page 18 of 100
Page 19 of 100
Page 20 of 100
Page 21 of 100
Page 22 of 100
Page 23 of 100
Page 24 of 100
Page 25 of 100
Page 26 of 100
Page 27 of 100
Page 28 of 100
Page 29 of 100
Page 30 of 100
Page 31 of 100
Page 32 of 100
Page 33 of 100
Page 34 of 100
Page 35 of 100
Page 36 of 100
Page 37 of 100
Page 38 of 100
Page 39 of 100
Page 40 of 100
Page 41 of 100
Page 42 of 100
Page 43 of 100
Page 44 of 100
Page 45 of 100
Page 46 of 100
Page 47 of 100
Page 48 of 100
Page 49 of 100
Page 50 of 100
Page 51 of 100
Page 52 of 100
Page 53 of 100
Page 54 of 100
Page 55 of 100
Page 56 of 100
Page 57 of 100
Page 58 of 100
Page 59 of 100
Page 60 of 100
Page 61 of 100
Page 62 of 100
Page 63 of 100
Page 64 of 100
Page 65 of 100
Page 66 of 100
Page 67 of 100
Page 68 of 100
Page 69 of 100
Page 70 of 100
Page 71 of 100
Page 72 of 100
Page 73 of 100
Page 74 of 100
Page 75 of 100
Page 76 of 100
Page 77 of 100
Page 78 of 100
Page 79 of 100
Page 80 of 100
Page 81 of 100
Page 82 of 100
Page 83 of 100
Page 84 of 100
Page 85 of 100
Page 86 of 100
Page 87 of 100
Page 88 of 100
Page 89 of 100
Page 90 of 100
Page 91 of 100
Page 92 of 100
Page 93 of 100
Page 94 of 100
Page 95 of 100
Page 96 of 100
Page 97 of 100
Page 98 of 100
Page 99 of 100
Page 100 of 100