Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Data] [CI] Add batch inference benchmarks #37283

Merged
merged 15 commits into from
Jul 25, 2023
Merged

[Data] [CI] Add batch inference benchmarks #37283

merged 15 commits into from
Jul 25, 2023

Conversation

amogkam
Copy link
Contributor

@amogkam amogkam commented Jul 11, 2023 •
edited
Loading

Adds the following benchmarks as release tests

  1. Image classification on 10 GB raw image data, 1 GPU
  2. Image classification on 10 GB parquet data, 1 GPU
  3. Image classification on 300 GB raw image data, 16 GPUs
  4. Image classification on 300 GB parquet data, 16 GPUs
  5. Image classification on 10 TB parquet data, 40 GPUs, weekly frequency

Closes #37345
Closes #28174
Closes #31796

Why are these changes needed?

Related issue number

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@amogkam
wip
bd9fe05 
Signed-off-by: amogkam <[email protected]>
@amogkam
add tests
685b6b3 
Signed-off-by: amogkam <[email protected]>
@amogkam
typo
08c6ae1 
Signed-off-by: amogkam <[email protected]>
@amogkam
move
c7aa355 
Signed-off-by: amogkam <[email protected]>
c21
c21 approved these changes Jul 11, 2023
View reviewed changes
Copy link
Contributor

@c21 c21 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LG w/ some minor comments.

release/air_tests/air_benchmarks/workloads/compute_hetero_10x10_aws.yaml Outdated Show resolved Hide resolved
release/air_tests/air_benchmarks/workloads/gpu_batch_prediction.py Outdated Show resolved Hide resolved
release/release_tests.yaml Outdated

# 10 GB image classification raw images with 1 GPU.
# 1 g4dn.4xlarge
- name: air_benchmark_torch_batch_prediction_1_gpu_10gb_parquet
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we still want to keep (10GB, 1GPU), given we already have (300GB, 16GPU) to run nightly?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it can help...it's a pretty cheap job cost wise and can help isolate whether regressions are occurring in the single node/single GPU case, or only in the multi node case. But I am open to removing it.

raulchen
raulchen reviewed Jul 12, 2023
View reviewed changes
release/BUILD Outdated Show resolved Hide resolved
@amogkam
comments
0ec1073 
Signed-off-by: amogkam <[email protected]>
@amogkam
fix
d38b933 
Signed-off-by: amogkam <[email protected]>
@amogkam
wip
8e878af 
Signed-off-by: amogkam <[email protected]>
@amogkam
wip
33fc6a5 
Signed-off-by: amogkam <[email protected]>
@c21
Copy link
Contributor

c21 commented Jul 21, 2023

@amogkam - is it good to merge?

@amogkam
Copy link
Contributor Author

amogkam commented Jul 22, 2023

@c21, no, let me kick off the release test

@amogkam
rename
f31be72 
Signed-off-by: amogkam <[email protected]>
@amogkam
Copy link
Contributor Author

amogkam commented Jul 24, 2023 •
edited
Loading

10 tb test passing with 11580.81978035109 images/second

https://buildkite.com/ray-project/release-tests-pr/builds/46310

@amogkam
bump timeout
14c7597 
Signed-off-by: amogkam <[email protected]>
@amogkam amogkam merged commit e1a97c3 into ray-project:master Jul 25, 2023
2 checks passed
@amogkam amogkam deleted the batch-inference-release-tests branch July 25, 2023 00:58
NripeshN pushed a commit to NripeshN/ray that referenced this pull request Aug 15, 2023
@amogkam @NripeshN
[Data] [CI] Add batch inference benchmarks (ray-project#37283)
d812616 
Adds the following benchmarks as release tests

Image classification on 10 GB raw image data, 1 GPU
Image classification on 10 GB parquet data, 1 GPU
Image classification on 300 GB raw image data, 16 GPUs
Image classification on 300 GB parquet data, 16 GPUs
Image classification on 10 TB parquet data, 40 GPUs, weekly frequency

---------

Signed-off-by: amogkam <[email protected]>
Signed-off-by: NripeshN <[email protected]>
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
@amogkam @harborn
[Data] [CI] Add batch inference benchmarks (ray-project#37283)
6061aa6 
Adds the following benchmarks as release tests

Image classification on 10 GB raw image data, 1 GPU
Image classification on 10 GB parquet data, 1 GPU
Image classification on 300 GB raw image data, 16 GPUs
Image classification on 300 GB parquet data, 16 GPUs
Image classification on 10 TB parquet data, 40 GPUs, weekly frequency

---------

Signed-off-by: amogkam <[email protected]>
Signed-off-by: harborn <[email protected]>
harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023
@amogkam @harborn
[Data] [CI] Add batch inference benchmarks (ray-project#37283)
31cfcb2 
Adds the following benchmarks as release tests

Image classification on 10 GB raw image data, 1 GPU
Image classification on 10 GB parquet data, 1 GPU
Image classification on 300 GB raw image data, 16 GPUs
Image classification on 300 GB parquet data, 16 GPUs
Image classification on 10 TB parquet data, 40 GPUs, weekly frequency

---------

Signed-off-by: amogkam <[email protected]>
arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023
@amogkam @arvind-chandra
[Data] [CI] Add batch inference benchmarks (ray-project#37283)
8bab1a9 
Adds the following benchmarks as release tests

Image classification on 10 GB raw image data, 1 GPU
Image classification on 10 GB parquet data, 1 GPU
Image classification on 300 GB raw image data, 16 GPUs
Image classification on 300 GB parquet data, 16 GPUs
Image classification on 10 TB parquet data, 40 GPUs, weekly frequency

---------

Signed-off-by: amogkam <[email protected]>
Signed-off-by: e428265 <[email protected]>
vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023
@amogkam
[Data] [CI] Add batch inference benchmarks (ray-project#37283)
2ea89a0 
Adds the following benchmarks as release tests

Image classification on 10 GB raw image data, 1 GPU
Image classification on 10 GB parquet data, 1 GPU
Image classification on 300 GB raw image data, 16 GPUs
Image classification on 300 GB parquet data, 16 GPUs
Image classification on 10 TB parquet data, 40 GPUs, weekly frequency

---------

Signed-off-by: amogkam <[email protected]>
Signed-off-by: Victor <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raulchen raulchen raulchen left review comments

@matthewdeng matthewdeng matthewdeng approved these changes

@c21 c21 c21 approved these changes

@richardliaw richardliaw Awaiting requested review from richardliaw

@gjoliver gjoliver Awaiting requested review from gjoliver

@krfricke krfricke Awaiting requested review from krfricke

@xwjiang2010 xwjiang2010 Awaiting requested review from xwjiang2010

@Yard1 Yard1 Awaiting requested review from Yard1

@maxpumperla maxpumperla Awaiting requested review from maxpumperla

Assignees

@raulchen raulchen

@c21 c21

Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@amogkam @c21 @raulchen @matthewdeng