Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

In env check, step only expected agents #26425

Merged

Conversation

mgerstgrasser
Copy link
Contributor

@mgerstgrasser mgerstgrasser commented Jul 9, 2022

Why are these changes needed?

The check_multiagent_environments() precheck right now calls env.step() with an action dict containing actions for all agents in the environment. For turn-based environments that don't expects all agents to act at the same time, this could break things.

Related issue number

Closes #26413

Checks

  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@mgerstgrasser
Copy link
Contributor Author

As part of this, the behavior of env.action_space_contains() needed to change, as it was returning False if only a subset of agents were contained in the sample action dict. That case now returns True if the action for each agent submitted is contained in that agent's action space. I notice that env.observation_space_contains() already handled that case for observations, but raised a warning. We could do that for env.action_space_contains() too, although the wording of the warning message is confusing IMHO. It says "your environment returns...", which assumes that these methods are only ever called on an observation / action dict returned from the environment. Maybe it would make more sense to check for "incomplete" observation dicts and raise a warning in check_multiagent_environments() instead of the environment methods?

@sven1977 sven1977 merged commit f0e9d1a into ray-project:master Jul 15, 2022
truelegion47 pushed a commit to truelegion47/ray that referenced this pull request Jul 16, 2022
* master: (35 commits)
  [data] Refactor all to all op implementations into a separate file (ray-project#26585)
  [Datasets] Explicitly define Dataset-like APIs in DatasetPipeline class (ray-project#26394)
  [Serve][Part2] Migrate the tests to use deployment graph api (ray-project#26507)
  [Serve] Default to EveryNode when starting Serve from REST API (ray-project#26588)
  Revert "[KubeRay][Autoscaler][Core] Add a flag to disable ray status version check (ray-project#26584)" (ray-project#26597)
  [air] Add initial benchmark section (ray-project#26608)
  [Workflow] Remove workflow execution module (ray-project#26504)
  [air] Add xgboost release test for silver tier(10-node case). (ray-project#26460)
  Revert "Revert "[serve] Use soft constraint for pinning controller on head node (ray-project#25091)" (ray-project#25857)" (ray-project#25858)
  [RLlib] Fixes MARWIL release tests (ray-project#26586)
  [Datasets] Improve read_xxx experience of HTTP file (ray-project#26454)
  Cleanup ActorContext due to multi actor instances got removed. (ray-project#26497)
  Print newest_ckpt_path when resuming trial. (ray-project#26561)
  Fix test_serialization_error_message for pytest 6.x (ray-project#26591)
  [RLlib] Make DQN update_target use only trainable variables. (ray-project#25226)
  [RLlib] In env check, step only expected agents. (ray-project#26425)
  [RLlib] `restart_failed_sub_environments` now works for MA cases and crashes during `reset()`; +more tests and logging; add eval worker sub-env fault tolerance test. (ray-project#26276)
  [runtime env] plugin refactor[4/n]: remove runtime env protobuf (ray-project#26522)
  Improve streaming read performance for default configuration. (ray-project#26587)
  [Dashboard] Fix test dashboard flaky by catch an expected exception (ray-project#26555)
  ...
truelegion47 pushed a commit to truelegion47/ray that referenced this pull request Jul 16, 2022
xwjiang2010 pushed a commit to xwjiang2010/ray that referenced this pull request Jul 19, 2022
avnishn pushed a commit to smorad/ray that referenced this pull request Jul 20, 2022
klwuibm pushed a commit to yuanchi2807/ray that referenced this pull request Jul 27, 2022
franklsf95 pushed a commit to franklsf95/ray that referenced this pull request Aug 2, 2022
gramhagen pushed a commit to gramhagen/ray that referenced this pull request Aug 15, 2022
gramhagen pushed a commit to gramhagen/ray that referenced this pull request Aug 15, 2022
Stefan-1313 pushed a commit to Stefan-1313/ray_mod that referenced this pull request Aug 18, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[RLlib] check_multiagent_environments() tries to step all agents at the same time
3 participants