Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

If sanity build is not found, wait until Jenkins recognizes it. #18119

Merged
merged 2 commits into from
Apr 21, 2020

Conversation

josephevans
Copy link
Contributor

Description

I ran into some problems when activating the staggered build pipeline in prod. It seems there is a slight delay (on a high-load server) between when Jenkins starts the full-build job and having the sanity job being recognized by Jenkins. This change allows it to retry until the job does exist.

@mxnet-bot
Copy link

Hey @josephevans , Thanks for submitting the PR
All tests are already queued to run once. If tests fail, you can trigger one or more tests again with the following commands:

  • To trigger all jobs: @mxnet-bot run ci [all]
  • To trigger specific jobs: @mxnet-bot run ci [job1, job2]

CI supported jobs: [windows-cpu, sanity, unix-gpu, edge, windows-gpu, website, clang, centos-cpu, unix-cpu, miscellaneous, centos-gpu]


Note:
Only following 3 categories can trigger CI :PR Author, MXNet Committer, Jenkins Admin.
All CI tests must pass before the PR can be merged.

@josephevans
Copy link
Contributor Author

@mxnet-bot run ci [unix-gpu]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [unix-gpu]

@josephevans
Copy link
Contributor Author

@mxnet-bot run ci [unix-gpu]

@mxnet-bot
Copy link

Jenkins CI successfully triggered : [unix-gpu]

Copy link
Contributor

@ChaiBapchya ChaiBapchya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the fix! LGTM!

@leezu leezu merged commit 84400fb into apache:master Apr 21, 2020
josephevans added a commit to josephevans/mxnet that referenced this pull request Apr 23, 2020
…he#18119)

* If sanity build is not found, wait until Jenkins recognizes it.

* Also add a timeout of 30m for sanity build to run and complete, so we don't get stuck in a loop.

Co-authored-by: Joe Evans <[email protected]>
josephevans added a commit to josephevans/mxnet that referenced this pull request Apr 23, 2020
…he#18119)

* If sanity build is not found, wait until Jenkins recognizes it.

* Also add a timeout of 30m for sanity build to run and complete, so we don't get stuck in a loop.

Co-authored-by: Joe Evans <[email protected]>
josephevans added a commit to josephevans/mxnet that referenced this pull request Apr 23, 2020
…he#18119)

* If sanity build is not found, wait until Jenkins recognizes it.

* Also add a timeout of 30m for sanity build to run and complete, so we don't get stuck in a loop.

Co-authored-by: Joe Evans <[email protected]>
leezu pushed a commit that referenced this pull request Apr 23, 2020
* For mxnet-validation pipeline, require sanity build to complete successfully before running other build pipelines. (#17999)

* Refactor staggered builds - create new full build pipeline that runs sanity check first, then starts all other builds.

* Move list of build jobs to top of file for clarity. Preserve whole job path in case we use nested folders in the future.

Co-authored-by: Joe Evans <[email protected]>

* If sanity build is not found, wait until Jenkins recognizes it. (#18119)

* If sanity build is not found, wait until Jenkins recognizes it.

* Also add a timeout of 30m for sanity build to run and complete, so we don't get stuck in a loop.

Co-authored-by: Joe Evans <[email protected]>

Co-authored-by: Joe Evans <[email protected]>
leezu pushed a commit that referenced this pull request Apr 24, 2020
* For mxnet-validation pipeline, require sanity build to complete successfully before running other build pipelines. (#17999)

* Refactor staggered builds - create new full build pipeline that runs sanity check first, then starts all other builds.

* Move list of build jobs to top of file for clarity. Preserve whole job path in case we use nested folders in the future.

Co-authored-by: Joe Evans <[email protected]>

* If sanity build is not found, wait until Jenkins recognizes it. (#18119)

* If sanity build is not found, wait until Jenkins recognizes it.

* Also add a timeout of 30m for sanity build to run and complete, so we don't get stuck in a loop.

Co-authored-by: Joe Evans <[email protected]>

Co-authored-by: Joe Evans <[email protected]>
leezu pushed a commit that referenced this pull request Apr 24, 2020
* For mxnet-validation pipeline, require sanity build to complete successfully before running other build pipelines. (#17999)

* Refactor staggered builds - create new full build pipeline that runs sanity check first, then starts all other builds.

* Move list of build jobs to top of file for clarity. Preserve whole job path in case we use nested folders in the future.

Co-authored-by: Joe Evans <[email protected]>

* If sanity build is not found, wait until Jenkins recognizes it. (#18119)

* If sanity build is not found, wait until Jenkins recognizes it.

* Also add a timeout of 30m for sanity build to run and complete, so we don't get stuck in a loop.

Co-authored-by: Joe Evans <[email protected]>

Co-authored-by: Joe Evans <[email protected]>
AntiZpvoh pushed a commit to AntiZpvoh/incubator-mxnet that referenced this pull request Jul 6, 2020
…he#18119)

* If sanity build is not found, wait until Jenkins recognizes it.

* Also add a timeout of 30m for sanity build to run and complete, so we don't get stuck in a loop.

Co-authored-by: Joe Evans <[email protected]>
@josephevans josephevans deleted the sanity_build_fixes branch January 22, 2021 01:04
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

5 participants