forked from babylm/babylm.github.io
-
Notifications
You must be signed in to change notification settings - Fork 0
/
index.html
83 lines (46 loc) · 4.09 KB
/
index.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
<!DOCTYPE html>
<html>
<head>
<link rel="stylesheet" href="stylesheet.css">
</head>
<body>
<div style="display:inline">
<img style="float: left; padding-right: 20px;" src="./images/pacifier.png" height="80">
<div class="master-title"> <b>Baby</b>LM Challenge </div>
<div class="subheader"> Sample-efficient pretraining on a developmentally plausible corpus </div>
</div>
<div id="navbar">
<h4> <a href="index.html"> Overview </a> • <a href="guidelines.html"> Guidelines </a> • <a href="timeline.html"> Timeline</a> • <a href="faqs.html"> FAQs </a> <hr> </h4>
</div>
<div class="greybox">
<div class="paragraph"> <b> Summary: </b> This shared task challenges community members to train a language model <b>from scratch</b> on the same amount of linguistic data available to a child. Submissions should be implemented in Huggingface's Transformers library and will be evaluated on a shared pipeline. This shared task is co-sponsored by <a href="https://cmclorg.github.io/">CMCL</a> and <a href="https://www.conll.org/">CoNLL</a>. </div>
<div class="bullet"> • <a href="https://github.com/babylm/babylm.github.io/raw/main/babylm_data.zip"; download="download">Download Dataset (700MB unzipped)</a></div>
<div class="bullet"> • Evaluation pipeline to be released <b> March 2023 </b> </div>
<div class="bullet"> • Results due by <b> July 15, 2023 </b> </div>
<div class="bullet"> • Paper submission due by <b> August 1, 2023</b> </div>
<div class="paragraph"> See the <a href="guidelines.html">guidelines</a> for an overview of submission tracks and pretraining data. See the <a href="https://arxiv.org/abs/2301.11796">call for papers</a> for a detailed description of the task setup and data. </div>
</div>
<div class="title"> Overview </div>
<img style="float: right; padding-left: 20px; padding-bottom: 15px;" src="./images/model_sizes.png" height="160">
<div class="paragraph"> Huge effort has been put into optimizing LM pretraining at massive scales in the last several years. While growing parameter counts often get the most attention, datasets have also grown by orders of magnitude. For example, <a href="https://arxiv.org/abs/2203.15556v1"> Chinchilla </a> sees 1.4 <b>trillion</b> words during training---well over 10000 words for every one word a 13 year old child has heard in their entire life.</div>
<div class="paragraph"> The goal of this shared task is to incentivize researchers with an interest in pretraining or cognitive modeling to focus their efforts on optimizing pretraining given data limitations inspired by human development. Additionally, we hope to democratize research on pretraining—which is typically thought to be practical only for large industry groups—by drawing attention to open problems that can be addressed on a university budget. </div>
<div class="title" > Why <100 Million Words? </div>
<div class="paragraph"> Focusing on scaled down pretraining has several potential benefits: First, small scale pretraining can be a sandbox for developing novel techniques for improving data efficiency. These techniques have the potential to then scale up to larger scales commonly seen in applied NLP or used to enhance current approaches to modeling low-resource languages. Second, improving our ability to train LMs on the same kinds and quantities of data that humans learn from hopefully will give us greater access to plausible cognitive models of humans and help us understand what allows humans to acquire language so efficiently. </div>
<div class="title"> Organization Team </div>
<div class = "people">
<div class="bullet">• Leshem Choshen </div>
<div class="bullet">• Ryan Cotterell </div>
<div class="bullet">• Kundan Krishna </div>
<div class="bullet">• Tal Linzen </div>
<div class="bullet">• Haokun Liu </div>
<div class="bullet">• Aaron Mueller </div>
<div class="bullet">• Alex Warstadt </div>
<div class="bullet">• Ethan Wilcox </div>
<div class="bullet">• Adina Williams </div>
<div class="bullet">• Chengxu Zhuang </div>
</div>
</div>
<div class="footer">
<div style="float:right;"> Images provided by Smashicons </div>
</div>
</body>