remove enwik8 data from repository

EleutherAI · Dec 26, 2020 · f376295 · f376295
1 parent f38c371
commit f376295
Show file tree

Hide file tree

Showing 4 changed files with 4 additions and 9 deletions.
diff --git a/.gitignore b/.gitignore
@@ -1,3 +1,6 @@
+## Data
+data/
+
 ## Python
 __pycache__/
 

diff --git a/README.md b/README.md
@@ -7,12 +7,6 @@ An implementation of model parallel GPT-3-like models on GPUs, based on the Deep
 $ pip install -r requirements.txt
 ```
 
-Test locally
-
-```bash
-$ python train_enwik8.py
-```
-
 Test deepspeed locally
 
 ```bash

diff --git a/data/README.md b/data/README.md
diff --git a/gpt_neox/utils.py b/gpt_neox/utils.py
@@ -13,6 +13,7 @@
 
 def prepare_enwik8_data():
  if not os.path.isfile('./data/enwik8.gz'):
+ os.system('mkdir -p ./data')
  os.system('wget http:https://eaidata.bmk.sh/data/enwik8.gz -O ./data/enwik8.gz')
 
  with gzip.open('./data/enwik8.gz') as file: