Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Feature] RLHF networks #1319

Merged
merged 4 commits into from
Jul 3, 2023
Merged

[Feature] RLHF networks #1319

merged 4 commits into from
Jul 3, 2023

Conversation

apbard
Copy link
Contributor

@apbard apbard commented Jun 27, 2023 •
edited
Loading

This PR adds a class method that builds an ActorValueOperator out of a base LMHeadModel-like model, it also adds a VmapModule wrapper useful to hide a batch dimension and satisfy the wrapped module

@facebook-github-bot facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jun 27, 2023
@apbard apbard changed the title [Feature ] Rlhf networks [Feature] RLHF networks Jun 27, 2023
@apbard apbard force-pushed the rlhf-networks branch 4 times, most recently from 066c231 to c67b4be Compare June 27, 2023 10:52
@apbard apbard mentioned this pull request Jun 27, 2023
Closed
vmoens
vmoens reviewed Jun 28, 2023
View reviewed changes
Copy link
Contributor

@vmoens vmoens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this

torchrl/modules/tensordict_module/actors.py Outdated Show resolved Hide resolved
torchrl/modules/tensordict_module/actors.py Outdated Show resolved Hide resolved
torchrl/modules/tensordict_module/actors.py Outdated Show resolved Hide resolved
torchrl/modules/tensordict_module/common.py Show resolved Hide resolved
test/test_actors.py Outdated Show resolved Hide resolved
torchrl/modules/tensordict_module/common.py Show resolved Hide resolved
@vmoens vmoens added the enhancement New feature or request label Jun 28, 2023
vmoens
vmoens approved these changes Jun 28, 2023
View reviewed changes
Copy link
Contributor

@vmoens vmoens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

torchrl/modules/tensordict_module/actors.py Outdated Show resolved Hide resolved
torchrl/modules/tensordict_module/common.py Outdated Show resolved Hide resolved
vmoens
vmoens approved these changes Jul 3, 2023
View reviewed changes
Copy link
Contributor

@vmoens vmoens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@vmoens vmoens merged commit a6d76e2 into pytorch:main Jul 3, 2023
38 of 55 checks passed
@vmoens vmoens deleted the rlhf-networks branch July 3, 2023 09:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@vmoens vmoens vmoens approved these changes

Assignees
No one assigned
Labels
CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. enhancement New feature or request
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@apbard @vmoens @facebook-github-bot