SyntaxError in preprocessing_data.py

[bpm246]

I'm trying to downloand and tokenize Enron Emails but when running preprocessing_data.py a Syntax Error pops up. Specifically in line 105

File "tools/preprocess_data.py", line 105
def yield_from_files(fnames: list, semaphore):
^
SyntaxError: invalid syntax

My python version is 3.8.5, so there shouldn't be any compatiblity problem.

Any ideas why is this happening?

[sdtblck]

I ran into this the other day - Are you running tools/preprocess_data.py or prepare_data.py in the main directory?

If the latter, I think it's because we're just using python to call preprocess_data.py here https://github.com/EleutherAI/gpt-neox/blob/main/tools/corpora.py#L133 and in some setups that can default to using python 2 - where yield from wasn't an operator yet.

I'll try to make this more robust in a future PR - but for now can you let me know if your problem is solved by changing the line linked above to call python3 rather than python?

[bpm246]

I tried to run the program with python3, but it kept getting the same error.
I wasn´t using anaconda as recommended by you, and there was the problem. I tried the process after installing anaconda and the necessary packages and it works without problem