Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add __del__ at Vllm for deleting model #10524

Closed
wants to merge 4 commits into from
Closed

Conversation

vkehfdl1
Copy link
Contributor

@vkehfdl1 vkehfdl1 commented Feb 8, 2024

Description

When using more than one Vllm model in single program, it occurs error. There is a vllm function destroy_model_parallel() for preventing this kind of error.

There is a description about this issue at here.

I tested use two Vllm instance in a single python file at CUDA environment. And it did great.
Without this code, it occurred error 'data parallel group is already initialized.'

Fixes # (issue)

Type of Change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

  • I stared at the code and made sure it makes sense
  • Plus, I tested it in my H100 instance with CUDA, and it worked well when I make two Vllm instance and run it.

Suggested Checklist:

  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • I have made corresponding changes to the documentation
  • I have added Google Colab support for the newly added notebooks.
  • My changes generate no new warnings
  • I have added tests that prove my fix is effective or that my feature works
  • New and existing unit tests pass locally with my changes
  • I ran make format; make lint to appease the lint gods

When using more than one Vllm model in single program, it occurs error. There is a vllm function `destroy_model_parallel()` for preventing this kind of error.
@dosubot dosubot bot added the size:XS This PR changes 0-9 lines, ignoring generated files. label Feb 8, 2024
@dosubot dosubot bot added size:S This PR changes 10-29 lines, ignoring generated files. and removed size:XS This PR changes 0-9 lines, ignoring generated files. labels Feb 9, 2024
@dosubot dosubot bot added the lgtm This PR has been approved by a maintainer label Feb 10, 2024
@hatianzhang
Copy link
Contributor

thanks @vkehfdl1 lgtm!

@logan-markewich
Copy link
Collaborator

@vkehfdl1 apologies, with v0.10.0 the project structure has changed slightly

I can take a stab at porting this PR in a bit, if you don't get to it first :)

@hatianzhang
Copy link
Contributor

address refactoring here #10733

@vkehfdl1 vkehfdl1 deleted the patch-2 branch February 15, 2024 20:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
lgtm This PR has been approved by a maintainer size:S This PR changes 10-29 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants