Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG]: AssertionError: Accumulation steps are not supported in stage 3 of ColossalAI #4847

Closed
sherlcok314159 opened this issue Oct 1, 2023 · 4 comments · May be fixed by #4849
Closed

[BUG]: AssertionError: Accumulation steps are not supported in stage 3 of ColossalAI #4847

sherlcok314159 opened this issue Oct 1, 2023 · 4 comments · May be fixed by #4849
Labels
bug Something isn't working

Comments

@sherlcok314159
Copy link

🐛 Describe the bug

When using gradient accumulation in colossal_gemini, it causes this error.

Example Script:

torchrun --standalone --nproc_per_node=4 train_sft.py \
    --pretrain "meta-llama/Llama-2-7b-hf" \
    --model 'llama' \
    --strategy colossalai_gemini \
    --save_path output/Coati-7B \
    --dataset yizhongw/self_instruct \
    --batch_size 8 \
    --accumulation_steps 8 \
    --lr 2e-5 \
    --max_datasets_size 512 \
    --max_epochs 1 \
    --grad_checkpoint

Environment

No response
@sherlcok314159 sherlcok314159 added the bug Something isn't working label Oct 1, 2023
@149189
Copy link

149189 commented Oct 1, 2023 •
edited
Loading

@sherlcok314159 Hey! I would Love to Work on this Issue

@Issues-translate-bot
Copy link

Bot detected the issue body's language is not English, translate it automatically. 👯👭🏻🧑‍🤝‍🧑👫🧑🏿‍🤝‍🧑🏻👩🏾‍🤝‍👨🏿👬🏿

@sherlcok314159

@sherlcok314159 sherlcok314159 mentioned this issue Oct 2, 2023
Open
10 tasks
@Fridge003
Copy link
Contributor

hi, we are developing gradient accumulation feature for gemini, please check the update in the following days~

@Fridge003
Copy link
Contributor

Fridge003 commented Oct 19, 2023 •
edited
Loading

Hi @sherlcok314159 , gradient accumulation for gemini has been implemented, usage can be found in this tutorial. If there is any bug please contact me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Fix installation errors and the example for limited resources sherlcok314159/ColossalAI
4 participants
@Fridge003 @Issues-translate-bot @sherlcok314159 @149189