Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Convert the huggingface checkpoint to GPT-Neox checkpoint #116

Closed
ZhiYuanZeng opened this issue Aug 9, 2023 · 2 comments
Closed

Convert the huggingface checkpoint to GPT-Neox checkpoint #116

ZhiYuanZeng opened this issue Aug 9, 2023 · 2 comments

Comments

@ZhiYuanZeng
Copy link

ZhiYuanZeng commented Aug 9, 2023 •
edited

I want to continue the pretraining based on the pretrained pythia checkpoints. But I don't know whether I can use the checkpoints from huggingface directly. How to convert the huggingface checkpoint to the checkpoints that can be loaded by GPT-Neox.
Can you can provide the raw checkpoints from GPT-Neox?
@ZhiYuanZeng
Copy link
Author

I have another question: do python-1b, python-12b, and python-20b share the same tokenizer?

@ZhiYuanZeng
Copy link
Author

A script of GPT-Neox has made it (https://github.com/EleutherAI/gpt-neox/blob/main/tools/convert_hf_to_sequential.py).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ZhiYuanZeng