Skip to the notes
← All winter school notes

Guilherme Penedo

FineWeb 2 — Multilingual Web Data at Scale

Download PDF ↓

HPLT × NLPL Winter School · Day 2 · 11 February 2025 · Scroll to read

About these notes

Lecture material on building a large multilingual pre-training corpus: data quality, filtering, deduplication, language identification, tokenization, experiments, and evaluation.

  1. Page 1 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 1 of 74
  2. Page 2 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 2 of 74
  3. Page 3 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 3 of 74
  4. Page 4 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 4 of 74
  5. Page 5 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 5 of 74
  6. Page 6 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 6 of 74
  7. Page 7 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 7 of 74
  8. Page 8 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 8 of 74
  9. Page 9 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 9 of 74
  10. Page 10 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 10 of 74
  11. Page 11 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 11 of 74
  12. Page 12 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 12 of 74
  13. Page 13 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 13 of 74
  14. Page 14 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 14 of 74
  15. Page 15 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 15 of 74
  16. Page 16 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 16 of 74
  17. Page 17 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 17 of 74
  18. Page 18 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 18 of 74
  19. Page 19 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 19 of 74
  20. Page 20 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 20 of 74
  21. Page 21 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 21 of 74
  22. Page 22 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 22 of 74
  23. Page 23 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 23 of 74
  24. Page 24 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 24 of 74
  25. Page 25 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 25 of 74
  26. Page 26 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 26 of 74
  27. Page 27 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 27 of 74
  28. Page 28 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 28 of 74
  29. Page 29 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 29 of 74
  30. Page 30 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 30 of 74
  31. Page 31 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 31 of 74
  32. Page 32 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 32 of 74
  33. Page 33 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 33 of 74
  34. Page 34 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 34 of 74
  35. Page 35 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 35 of 74
  36. Page 36 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 36 of 74
  37. Page 37 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 37 of 74
  38. Page 38 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 38 of 74
  39. Page 39 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 39 of 74
  40. Page 40 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 40 of 74
  41. Page 41 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 41 of 74
  42. Page 42 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 42 of 74
  43. Page 43 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 43 of 74
  44. Page 44 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 44 of 74
  45. Page 45 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 45 of 74
  46. Page 46 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 46 of 74
  47. Page 47 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 47 of 74
  48. Page 48 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 48 of 74
  49. Page 49 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 49 of 74
  50. Page 50 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 50 of 74
  51. Page 51 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 51 of 74
  52. Page 52 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 52 of 74
  53. Page 53 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 53 of 74
  54. Page 54 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 54 of 74
  55. Page 55 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 55 of 74
  56. Page 56 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 56 of 74
  57. Page 57 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 57 of 74
  58. Page 58 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 58 of 74
  59. Page 59 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 59 of 74
  60. Page 60 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 60 of 74
  61. Page 61 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 61 of 74
  62. Page 62 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 62 of 74
  63. Page 63 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 63 of 74
  64. Page 64 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 64 of 74
  65. Page 65 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 65 of 74
  66. Page 66 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 66 of 74
  67. Page 67 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 67 of 74
  68. Page 68 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 68 of 74
  69. Page 69 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 69 of 74
  70. Page 70 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 70 of 74
  71. Page 71 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 71 of 74
  72. Page 72 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 72 of 74
  73. Page 73 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 73 of 74
  74. Page 74 of 74 from FineWeb 2 — Multilingual Web Data at Scale, included in Lucia Domenichelli's HPLT × NLPL Winter School notes.
    Page 74 of 74