Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

skipping tests that cannot fit in nightly CI machine #17450

Merged
merged 1 commit into from
Feb 5, 2020

Conversation

access2rohit
Copy link
Contributor

@access2rohit access2rohit commented Jan 27, 2020

Description

skipping following tests as they require memory more than present in nightly CI.
A Stacked run of tests of these operators along with other ops doesn't free up memory on time resulting in Memory Error on nightly CI machine. Also updates system path for correct imports.

Fixes #14981
Fixes #14980

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

  • Changes are complete (i.e. I finished coding on this PR)
  • All changes have test coverage:
  • Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
  • Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
  • Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
  • To the best of my knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

Tests

ci/build.py --docker-registry mxnetci --nvidiadocker --platform ubuntu_nightly_gpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh nightly_test_large_tensor

2020-01-28 01:08:03,925 - root - INFO - Started container: 5bceca2b8f26
+ NOSE_COVERAGE_ARGUMENTS='--with-coverage --cover-inclusive --cover-xml --cover-branches --cover-package=mxnet'
+ NOSE_TIMER_ARGUMENTS='--with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error'
+ CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'
+ CI_CMAKE_CUDA_ARCH='5.2 7.0'
+ set +x
+ export PYTHONPATH=./python/
+ PYTHONPATH=./python/
+ export DMLC_LOG_STACK_TRACE_DEPTH=10
+ DMLC_LOG_STACK_TRACE_DEPTH=10
+ nosetests-3.4 tests/nightly/test_large_array.py:test_tensor
S
----------------------------------------------------------------------
Ran 1 test in 125.654s

OK (SKIP=1)
+ nosetests-3.4 tests/nightly/test_large_array.py:test_nn
[01:15:48] src/executor/graph_executor.cc:2062: Subgraph backend MKLDNN is activated.
[01:15:52] src/executor/graph_executor.cc:2062: Subgraph backend MKLDNN is activated.
S
----------------------------------------------------------------------
Ran 1 test in 344.892s

OK (SKIP=1)
+ nosetests-3.4 tests/nightly/test_large_array.py:test_basic
S
----------------------------------------------------------------------
Ran 1 test in 156.411s

OK (SKIP=1)

ci/build.py --docker-registry mxnetci --nvidiadocker --platform ubuntu_nightly_gpu --docker-build-retries 3 --shm-size 500m /work/runtime_functions.sh nightly_test_large_vector

+ NOSE_TIMER_ARGUMENTS='--with-timer --timer-ok 1 --timer-warning 15 --timer-filter warning,error'
+ CI_CUDA_COMPUTE_CAPABILITIES='-gencode=arch=compute_52,code=sm_52 -gencode=arch=compute_70,code=sm_70'
+ CI_CMAKE_CUDA_ARCH='5.2 7.0'
+ set +x
+ export PYTHONPATH=./python/
+ PYTHONPATH=./python/
+ export DMLC_LOG_STACK_TRACE_DEPTH=10
+ DMLC_LOG_STACK_TRACE_DEPTH=10
+ nosetests-3.4 tests/nightly/test_large_vector.py:test_tensor
S
----------------------------------------------------------------------
Ran 1 test in 107.439s

OK (SKIP=1)
+ nosetests-3.4 tests/nightly/test_large_vector.py:test_nn
[06:22:40] src/executor/graph_executor.cc:1982: Subgraph backend MKLDNN is activated.
[06:27:34] src/executor/graph_executor.cc:1982: Subgraph backend MKLDNN is activated.
.
----------------------------------------------------------------------
Ran 1 test in 653.536s

OK
+ nosetests-3.4 tests/nightly/test_large_vector.py:test_basic
S
----------------------------------------------------------------------
Ran 1 test in 65.930s

OK (SKIP=1)

@access2rohit
Copy link
Contributor Author

@mxnet-label-bot add [pr-awaiting-review]

@lanking520 lanking520 added the pr-awaiting-review PR is waiting for code review label Jan 28, 2020
@access2rohit
Copy link
Contributor Author

@apeforest @ChaiBapchya PR ready for review

@@ -689,6 +693,7 @@ def check_pick():
res = mx.nd.pick(a, b)
assert res.shape == b.shape

@unittest.skip("Memory doesn't free up after stacked execution with other ops, tracked at https://github.com/apache/incubator-mxnet/issues/17411")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this line is too long. please break it to meet PEP8 style

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

aah ... got it ! Will do

Copy link
Contributor

@apeforest apeforest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please do lint

Copy link
Contributor

@apeforest apeforest left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@access2rohit
Copy link
Contributor Author

@mxnet-label-bot update [pr-awaiting-merge]

@lanking520 lanking520 added pr-awaiting-merge Review and CI is complete. Ready to Merge and removed pr-awaiting-review PR is waiting for code review labels Jan 29, 2020
Copy link
Contributor

@ChaiBapchya ChaiBapchya left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!
Retriggered windows-gpu for you. Hopefully it passes!

@apeforest apeforest merged commit 9f35376 into apache:master Feb 5, 2020
zheyuye pushed a commit to zheyuye/incubator-mxnet that referenced this pull request Feb 19, 2020
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
pr-awaiting-merge Review and CI is complete. Ready to Merge
Projects
None yet
4 participants