-
Notifications
You must be signed in to change notification settings - Fork 238
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Unable to reproduce result on HalfCheetah-v2 #75
Comments
Thanks a lot for reporting this! This is not expected, and it's not immediately obvious to me what might cause it. Just to make sure: are all these figures produced with the default values in the |
yup, the figures are produced with the default values in variants.py. The results in each figure are averaged over 5 random initial seeds. |
Thanks! I'll try to look into this soon. |
Thank you for your comment! I will rerun master and let you know how it goes here. |
Thank you for your effort in maintaining the repo. It has been super useful and illuminating to read the code. Also, thank you for rerunning the master and helps me get to the bottom of my issue. Please find my results below: Running the master on 5th May, which corresponds to commit 1f6686d does not reproduce the reported results. This is because in my runs, 1 out of 5 is significantly worse than the other runs. The performance graph is shown below. However, you might not be able to reproduce this result by setting the initial seeds to the values in my runs. This is because of this issue #80. After I also set the seed of the environment and fix the value for the initial seeds in the runs, I obtained similar results where some of the runs are significantly worse than others. The output of pip freeze can be found in the file below I am using mujoco 1.50 for Linux. |
Thanks for the detailed response! This still seems a little odd since I'm not able to get any seeds failing like that. One difference I see in your |
I also observed very similar results as @quanvuong did (including runs that converged at ~2k return). I used mujoco 2.0 and mujoco-py 2.0.2.0. |
@Nicolinho would you mind pasting the output of your |
Hey @quanvuong and @Nicolinho, thanks a lot for providing the information. I spent a bit of time debugging this today. I couldn't find anything obvious that could be wrong, but was eventually able to reproduce this issue on one of my machines where 2/4 seeds failed. The weird thing is that still on other machines, the cheetah runs completely fine; I ran a total of 30 seeds on 3 different machines and none of them had the issue. 10 of those seeds are found from a comment in the latest PR: #85 (comment). I'll try to dig deeper soon and let you know if I find anything. |
It was so satisfying to finally know why this happened. Thank you for the investigative work! |
Hi @hartikainen , for the curve of one single trial, do you use smoothed curve? Thanks! |
I think the problem is that Gym has two separate random seeds for an environment. One is set during the creation of the environment. The other is the seed of I run some SAC experiments with a current gym version and first set the action space seed to zero by adding the following line to the
The result after 1 mil timesteps was similar to those shown above (for HalfCheetah-v2): Then I rerun the experiment with a random action space seed:
It made 3 of the 9 runs get stuck at the back-flip policy: This problem seems to be gone when I increase the amount of random steps in the beginning of training ( Gym commit: 98917cf |
I was able to readily reproduce the paper's results on HalfCheetah-v2 without changing any defaults: MuJoCo 2.0 |
Nice, thanks a ton for reporting these @xanderdunn! Seems like this issue can be closed now. |
* Updated to match new priority methods in CARES RL * latest for using PER buffer with new CARES RL * fixed image_wrapper to have sample function * fixed seed setting for openai gym * Updated seed for action space to (0) based on similar issues here: rail-berkeley/softlearning#75 * revert seed 0
I am unable to obtain the result as reported in the paper on the openai environment HalfCheetah-v2. The commit used to obtain this result is 1f6147c, which isn't too long ago. The result is averaged over 5 random initial seeds.
Do you know what might be causing this issue? Thank you!
I am able to obtain the result as reported (or close to it) in the paper on the remaining environments, posted here for reference.
The text was updated successfully, but these errors were encountered: