Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix issue: cuda index incorrectly -1 when tp_size>1 #496

Conversation

ZX-ModelCloud
Copy link
Contributor

vllm now needs to call the init_distributed_environment method correctly so that _LOCAL_RANK will not be -1 when tp_size>1.

vllm source code: https://github.com/vllm-project/vllm/blob/main/vllm/distributed/parallel_state.py#L193

Someone has reproduced this issue: #487 (comment)

@Qubitium
Copy link
Contributor

Qubitium commented Jun 3, 2024

Tested with vllm 0.4.3. No issues.

@merrymercy merrymercy mentioned this pull request Jun 7, 2024
@merrymercy
Copy link
Contributor

@Qubitium @ZhouXingg Thanks for the contributions. They are merged in #511.

@merrymercy merrymercy closed this Jun 7, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants