Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Parallelize probe training across layers #64

Closed
norabelrose opened this issue Feb 15, 2023 · 3 comments · Fixed by #86
Closed

Parallelize probe training across layers #64

norabelrose opened this issue Feb 15, 2023 · 3 comments · Fixed by #86
Assignees
@anshradh
Labels
enhancement New feature or request good first issue Good for newcomers
Milestone
PyPI 0.1 Release

Comments

@norabelrose
Copy link
Member

Instead of using torch.distributed to paralleize probe training, we can simply create a multiprocessing.Queue with one worker per GPU, and train the probes for each layer of a transformer in parallel.
@norabelrose norabelrose added enhancement New feature or request good first issue Good for newcomers labels Feb 15, 2023
@norabelrose
Copy link
Member Author

To be clear: a fix for this issue should remove all the vestigial calls to torch.distributed in the training code. It's not actually needed if we go this route.

@norabelrose norabelrose added this to the PyPI 0.1 Release milestone Feb 15, 2023
@anshradh
Copy link

I'd be interested in picking this up! Is there a preferred timeframe in which this gets done? My guess is that I could probably finish this over the coming weekend, but if that's not soon enough probably someone else should tackle it.

@norabelrose
Copy link
Member Author

@anshradh Yeah if it's done by the end of the weekend that should be fine! Feel free to ping me on the eliciting-latent-knowledge channel of the Eleuther Discord with questions

@anshradh anshradh mentioned this issue Feb 17, 2023
Closed
@norabelrose norabelrose mentioned this issue Feb 18, 2023
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@anshradh anshradh

Labels
enhancement New feature or request good first issue Good for newcomers
Projects
None yet
Milestone
PyPI 0.1 Release
Development

Successfully merging a pull request may close this issue.

Train reporters for different layers in parallel EleutherAI/elk
2 participants
@anshradh @norabelrose