Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Detect regression based on dates #1609

Closed
wants to merge 14 commits into from
Closed

Detect regression based on dates #1609

wants to merge 14 commits into from

Conversation

xuzhao9
Copy link
Contributor

@xuzhao9 xuzhao9 commented May 2, 2023 •
edited
Loading

Support using --start_date and --end_date to detect regressions by using files on Amazon S3.

Test plan (Need to set AWS credentials):

python regression_detector.py --platform gcp_a100 --end-date 2023-05-02 --name torch-nightly

control_env:
  pytorch_git_version: 521b386e57f357386caa821adb0d9bae140f272e
treatment_env:
  pytorch_git_version: f38a04a5c5ce632ba8b0534e3689e89f98e89d73
bisection: null
details:
  test_eval[phlippe_resnet-cuda-eager]_latency:
    control: 2.978329
    treatment: 3.53091
    delta: 0.18553390172811668
  test_eval[timm_resnest-cuda-eager]_gmem:
    control: 4.0126953125
    treatment: 4.5927734375
    delta: 0.14456072036991968
control_only_metrics: {}
treatment_only_metrics: {}


TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 521b386e57f357386caa821adb0d9bae140f272e

Affected PyTorch commit: f38a04a5c5ce632ba8b0534e3689e89f98e89d73

Affected Tests:
- test_eval[phlippe_resnet-cuda-eager]_latency: +0.18553%
- test_eval[timm_resnest-cuda-eager]_gmem: +0.14456%


Tests that were no longer run on affected commit:


Tests that were newly added on affected commit:


Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: No URL found, please look for the failing action in https://github.com/pytorch/benchmark/actions

cc @xuzhao9

@xuzhao9 xuzhao9 requested a review from janeyx99 May 2, 2023 14:45
@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

janeyx99
janeyx99 reviewed May 2, 2023
View reviewed changes
regression_detector.py Outdated Show resolved Hide resolved
janeyx99
janeyx99 reviewed May 2, 2023
View reviewed changes
regression_detector.py Outdated Show resolved Hide resolved
janeyx99
janeyx99 reviewed May 2, 2023
View reviewed changes
regression_detector.py Outdated Show resolved Hide resolved
janeyx99
janeyx99 reviewed May 2, 2023
View reviewed changes
regression_detector.py Outdated Show resolved Hide resolved
janeyx99
janeyx99 reviewed May 2, 2023
View reviewed changes
regression_detector.py Outdated Show resolved Hide resolved
janeyx99
janeyx99 reviewed May 2, 2023
View reviewed changes
userbenchmark/utils.py Outdated Show resolved Hide resolved
janeyx99
janeyx99 reviewed May 2, 2023
View reviewed changes
regression_detector.py Outdated Show resolved Hide resolved
@janeyx99
Copy link
Contributor

janeyx99 commented May 2, 2023

What's the test plan?

janeyx99
janeyx99 reviewed May 2, 2023
View reviewed changes
userbenchmark/utils.py Outdated Show resolved Hide resolved
@xuzhao9
Copy link
Contributor Author

xuzhao9 commented May 2, 2023

What's the test plan?

Updated, I still have some S3 issue running it locally though.

@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

janeyx99
janeyx99 approved these changes May 2, 2023
View reviewed changes
Copy link
Contributor

@janeyx99 janeyx99 left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks okay if passes local S3 expectations after addressing my comment about the reverse order traversal

@facebook-github-bot
Copy link
Contributor

@xuzhao9 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator.

@xuzhao9
Copy link
Contributor Author

xuzhao9 commented May 2, 2023

Looks okay if passes local S3 expectations after addressing my comment about the reverse order traversal

Thanks! I have fixed that issue.

@xuzhao9
Copy link
Contributor Author

xuzhao9 commented May 3, 2023

@janeyx99 Can I get a stamp on the Diff? Thanks!

@janeyx99
Copy link
Contributor

janeyx99 commented May 3, 2023

Done!

@facebook-github-bot
Copy link
Contributor

@xuzhao9 merged this pull request in aa2a190.

@xuzhao9 xuzhao9 deleted the xz9/v3-detector branch May 3, 2023 16:32
gairgeio added a commit to gairgeio/benchmark that referenced this pull request Aug 2, 2024
@gairgeio @facebook-github-bot
Detect regression based on dates (#1609)
6068b01 
Summary:
Support using `--start_date` and `--end_date` to detect regressions by using files on Amazon S3.

Test plan (Need to set AWS credentials):
```
python regression_detector.py --platform gcp_a100 --end-date 2023-05-02 --name torch-nightly
```
```
control_env:
  pytorch_git_version: 521b386e57f357386caa821adb0d9bae140f272e
treatment_env:
  pytorch_git_version: f38a04a5c5ce632ba8b0534e3689e89f98e89d73
bisection: null
details:
  test_eval[phlippe_resnet-cuda-eager]_latency:
    control: 2.978329
    treatment: 3.53091
    delta: 0.18553390172811668
  test_eval[timm_resnest-cuda-eager]_gmem:
    control: 4.0126953125
    treatment: 4.5927734375
    delta: 0.14456072036991968
control_only_metrics: {}
treatment_only_metrics: {}
```

```

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 521b386e57f357386caa821adb0d9bae140f272e

Affected PyTorch commit: f38a04a5c5ce632ba8b0534e3689e89f98e89d73

Affected Tests:
- test_eval[phlippe_resnet-cuda-eager]_latency: +0.18553%
- test_eval[timm_resnest-cuda-eager]_gmem: +0.14456%

Tests that were no longer run on affected commit:

Tests that were newly added on affected commit:

Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: No URL found, please look for the failing action in https://github.com/pytorch/benchmark/actions

cc xuzhao9
```

Pull Request resolved: pytorch/benchmark#1609

Reviewed By: janeyx99

Differential Revision: D45484697

Pulled By: xuzhao9

fbshipit-source-id: cd0b902c6f09bbf813f0e1c8fac07f6f148e61dd
bestappsandcodereviews7 added a commit to bestappsandcodereviews7/benchmark that referenced this pull request Aug 16, 2024
@bestappsandcodereviews7 @facebook-github-bot
Detect regression based on dates (#1609)
a9d88d0 
Summary:
Support using `--start_date` and `--end_date` to detect regressions by using files on Amazon S3.

Test plan (Need to set AWS credentials):
```
python regression_detector.py --platform gcp_a100 --end-date 2023-05-02 --name torch-nightly
```
```
control_env:
  pytorch_git_version: 521b386e57f357386caa821adb0d9bae140f272e
treatment_env:
  pytorch_git_version: f38a04a5c5ce632ba8b0534e3689e89f98e89d73
bisection: null
details:
  test_eval[phlippe_resnet-cuda-eager]_latency:
    control: 2.978329
    treatment: 3.53091
    delta: 0.18553390172811668
  test_eval[timm_resnest-cuda-eager]_gmem:
    control: 4.0126953125
    treatment: 4.5927734375
    delta: 0.14456072036991968
control_only_metrics: {}
treatment_only_metrics: {}
```

```

TorchBench CI has detected a performance signal or runtime regression.

Base PyTorch commit: 521b386e57f357386caa821adb0d9bae140f272e

Affected PyTorch commit: f38a04a5c5ce632ba8b0534e3689e89f98e89d73

Affected Tests:
- test_eval[phlippe_resnet-cuda-eager]_latency: +0.18553%
- test_eval[timm_resnest-cuda-eager]_gmem: +0.14456%

Tests that were no longer run on affected commit:

Tests that were newly added on affected commit:

Runtime regressions found?
No runtime errors were found in the new benchmarks run--you are all good there!

GitHub workflow that triggered this issue: No URL found, please look for the failing action in https://github.com/pytorch/benchmark/actions

cc xuzhao9
```

Pull Request resolved: pytorch/benchmark#1609

Reviewed By: janeyx99

Differential Revision: D45484697

Pulled By: xuzhao9

fbshipit-source-id: cd0b902c6f09bbf813f0e1c8fac07f6f148e61dd
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@janeyx99 janeyx99 janeyx99 approved these changes

Assignees
No one assigned
Labels
cla signed Merged
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@xuzhao9 @facebook-github-bot @janeyx99