Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[rllib] Basic port of baselines/deepq to rllib #709

Merged
merged 27 commits into from
Jul 7, 2017

Conversation

ericl
Copy link
Contributor

@ericl ericl commented Jul 5, 2017

This is a straightforward adaptation of the baselines DQN implementation to conform to the RLlib API. Files to pay attention to are rllib/dqn/dqn.py and rllib/dqn/example.py; the rest were mostly copied with linter fixes only.

I also fixed up the licensing here by appending the OpenAI MIT license to the top-level LICENSE file.

I have a couple ideas on how to parallelize this with Ray in a followup PR:

  • First, we can parallelize rollouts, however to preserve algorithm semantics this requires train_freq to be large enough to allow sufficient parallelism between training steps. Increasing train_freq will probably also require an equivalent increase of batch_size.
  • Second, we can parallelize the optimization step. This also requires the batch_size parameter to be increased. We might also consider multiple steps of optimization over replay buffer samples, similar to policy gradient.

There is also literature on parallelizing DQN in other ways but that might be out of scope for now.

On a GPU instance the Pong example spends about equal time in training and rollouts, so both could be potentially valuable.

cc @pcmoritz @royf

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/1177/
Test PASSed.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/1195/
Test PASSed.

@AmplabJenkins
Copy link

Merged build finished. Test PASSed.

@AmplabJenkins
Copy link

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/Ray-PRB/1199/
Test PASSed.

@pcmoritz pcmoritz merged commit f012e59 into ray-project:master Jul 7, 2017
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

3 participants