[air/output] Use Tune reporter if Tuner/tune.run was used

[krfricke]

Why are these changes needed?

Currently we use the number of samples to detect if we should use the Train output reporter or the tune output reporter.

However, this is confusing when developing on a small scale, e.g. a tuning run with only 1 trial, before going to multiple trials. Instead, we should use the entrypoint to decide which reporter we use. The Trainer will use the Train reporter, and Tuner, tune.run, and tune.run_experiments will use the tune reporter.

Related issue number

Checks

• I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
  • I've added any new APIs to the API Reference. For example, if I added a
    method in Tune, I've added it in doc/source/tune/api/ under the
    corresponding .rst file.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[scottsun94]

This change makes sense!

One question:
how about Rllib? As long as it doesn't use tuner, tune.run, and tune.run_experiments, it should just use trainer?

[justinvyu]

This change makes sense!

One question: how about Rllib? As long as it doesn't use tuner, tune.run, and tune.run_experiments, it should just use trainer?

@scottsun94 RLlib can only run through the Tune entrypoints. Trainer is from Ray Train.
Training an RLlib Algorithm outside of Tune doesn't come with reporting capabilities.

[justinvyu]

Makes sense! Are the failing tests unrelated?

[krfricke]

Makes sense! Are the failing tests unrelated?

Yes, they're due to a breakage that was fixed in #36157. I'll merge master to reflect this in the CI.

[scottsun94]

This change makes sense!

One question: how about Rllib? As long as it doesn't use tuner, tune.run, and tune.run_experiments, it should just use trainer?

@scottsun94 RLlib can only run through the Tune entrypoints. Trainer is from Ray Train.

Training an RLlib Algorithm outside of Tune doesn't come with reporting capabilities.

This makes sense. But what's the story of output for rllib (training an RLlib Algorithm outside of Tune)? Rllib needs to build its own output reporter?

[krfricke]

This change makes sense!

One question: how about Rllib? As long as it doesn't use tuner, tune.run, and tune.run_experiments, it should just use trainer?

@scottsun94 RLlib can only run through the Tune entrypoints. Trainer is from Ray Train.
Training an RLlib Algorithm outside of Tune doesn't come with reporting capabilities.

This makes sense. But what's the story of output for rllib (training an RLlib Algorithm outside of Tune)? Rllib needs to build its own output reporter?

Yes, if rllib is not using tune/air for execution, they have to implement their own reporting logic. They can potentially still use the same output reporter, but they have to call the hooks themselves.

The logic from this PR would then still use the train (single run) reporter, so it should be unaffected by the changes in this PR.

[scottsun94]

This change makes sense!

One question: how about Rllib? As long as it doesn't use tuner, tune.run, and tune.run_experiments, it should just use trainer?

@scottsun94 RLlib can only run through the Tune entrypoints. Trainer is from Ray Train.
Training an RLlib Algorithm outside of Tune doesn't come with reporting capabilities.

This makes sense. But what's the story of output for rllib (training an RLlib Algorithm outside of Tune)? Rllib needs to build its own output reporter?

Yes, if rllib is not using tune/air for execution, they have to implement their own reporting logic. They can potentially still use the same output reporter, but they have to call the hooks themselves.

The logic from this PR would then still use the train (single run) reporter, so it should be unaffected by the changes in this PR.

It makes sense. Thanks!