Static build instruction for MXNet in general #13914

lanking520 · 2019-01-17T01:51:27Z

Description

Adding instructions for Static build and a static build test stage in CI. Also make the mkldnn as a default on Maven publish and enable signal_handler for segfault.

@frankfliu @szha @marcoabreu @zachgk @gigasquid

Checklist

Essentials

Please feel free to remove inapplicable items for your PR.

The PR title starts with [MXNET-$JIRA_ID], where $JIRA_ID refers to the relevant JIRA issue created (except PRs with tiny changes)
Changes are complete (i.e. I finished coding on this PR)
All changes have test coverage:
Unit tests are added for small changes to verify correctness (e.g. adding a new operator)
Nightly tests are added for complicated/long-running ones (e.g. changing distributed kvstore)
Build tests will be added for build configuration changes (e.g. adding a new build option with NCCL)
Code is well-documented:
For user-facing API changes, API doc string has been updated.
For new C++ functions in header files, their functionalities and arguments are documented.
For new examples, README.md is added to explain the what the example does, the source of the dataset, expected performance on test set and reference to the original paper if applicable
Check the API doc at http:https://mxnet-ci-doc.s3-accelerate.dualstack.amazonaws.com/PR-$PR_ID/$BUILD_ID/index.html
To the my best knowledge, examples are either not affected by this change, or have been fixed to be compatible with this change

pengzhao-intel · 2019-01-17T03:36:18Z

Thanks @lanking520 to enable MKLDNN backend for Maven build.

Feel free to ping me if anything we can help.

stu1130 · 2019-01-17T18:56:45Z

@mxnet-label-bot add [pr-awaiting-review]
Thanks @lanking520

tools/staticbuild/README.md

ci/jenkins/Jenkins_steps.groovy

andrewfayres

LGTM

aaronmarkham · 2019-01-18T00:25:10Z

ci/publish/README.md

@@ -0,0 +1,33 @@
+# MXNet Publish Settings
+
+This folder contains the configuration of restricted node on Jenkins for the publish. It also contains a folder called `scala` that contains everything required for scala publish. In this `README`, we would bring a brief walkthrough of the Jenkins configuration as well as the usages of the scala deployment files.


Suggested change

This folder contains the configuration of restricted node on Jenkins for the publish. It also contains a folder called `scala` that contains everything required for scala publish. In this `README`, we would bring a brief walkthrough of the Jenkins configuration as well as the usages of the scala deployment files.

This folder contains the configuration for restricted nodes on Jenkins for the publishing MXNet artifacts. It also contains a folder called `scala` that contains everything required for publishing to Maven. In this `README`, we provide a brief walkthrough of the Jenkins configuration as well as the usage of the Scala deployment files. Python publishing is TBD.

aaronmarkham · 2019-01-18T00:27:08Z

ci/publish/README.md

+This folder contains the configuration of restricted node on Jenkins for the publish. It also contains a folder called `scala` that contains everything required for scala publish. In this `README`, we would bring a brief walkthrough of the Jenkins configuration as well as the usages of the scala deployment files.
+
+## Jenkins
+Currently, Jenkins contains three build stages, namely `Build Packages`, `Test Packages` and `Deploy Packages`. During the `build package` stages, all dependencies will be built and a Scala package would be created. In the second stage, the package created from the previous stage would move to this stage to specifically run the tests. In the final stage, the packages passed the test would be deployed by the instances.


Suggested change

Currently, Jenkins contains three build stages, namely `Build Packages`, `Test Packages` and `Deploy Packages`. During the `build package` stages, all dependencies will be built and a Scala package would be created. In the second stage, the package created from the previous stage would move to this stage to specifically run the tests. In the final stage, the packages passed the test would be deployed by the instances.

Currently, Jenkins contains three build stages, namely `Build Packages`, `Test Packages` and `Deploy Packages`. During the `build package` stages, all dependencies are built and a Scala package are created. In the second stage, the package created from the previous stage moves to this stage to specifically run the tests. In the final stage, the packages that pass the tests are deployed by the instances.

aaronmarkham · 2019-01-18T00:28:16Z

ci/publish/README.md

+
+Apart from that, the script used to create the environment and publish are available under `ci/docker/install`:
+
+- `ubuntu_publish.sh` install all required dependencies for Ubuntu 14.04 for publishing


Suggested change

- `ubuntu_publish.sh` install all required dependencies for Ubuntu 14.04 for publishing

- `ubuntu_publish.sh` installs all required dependencies for Ubuntu 14.04 for publishing

aaronmarkham · 2019-01-18T00:28:27Z

ci/publish/README.md

+Apart from that, the script used to create the environment and publish are available under `ci/docker/install`:
+
+- `ubuntu_publish.sh` install all required dependencies for Ubuntu 14.04 for publishing
+- `ubuntu_base.sh` install minimum dependencies required to run the published packages


Suggested change

- `ubuntu_base.sh` install minimum dependencies required to run the published packages

- `ubuntu_base.sh` installs minimum dependencies required to run the published packages

aaronmarkham · 2019-01-18T00:29:12Z

ci/publish/README.md

+- `ubuntu_base.sh` install minimum dependencies required to run the published packages
+
+## Scala publish
+Currently Scala publish on Linux is fully supported on jenkins. The `scala/` folder contains all files needed to do the publish. Here is a breif instroduction of the files:


Suggested change

Currently Scala publish on Linux is fully supported on jenkins. The `scala/` folder contains all files needed to do the publish. Here is a breif instroduction of the files:

Currently Scala publish on Linux is fully supported on Jenkins. The `scala/` folder contains all files needed for publishing. Here is a brief introduction of the files:

aaronmarkham · 2019-01-18T01:11:35Z

tools/staticbuild/README.md

@@ -0,0 +1,32 @@
+# MXNet Static build
+
+This folder contains the core script used to build the static library. This README would bring you the information and usages of the script in here. Please be aware, all of the scripts are designed to be run under the root folder.


Suggested change

This folder contains the core script used to build the static library. This README would bring you the information and usages of the script in here. Please be aware, all of the scripts are designed to be run under the root folder.

This folder contains the core script used to build the static library. This README provides information on how to use the scripts in this folder. Please be aware, all of the scripts are designed to be run under the root folder.

aaronmarkham · 2019-01-18T01:13:06Z

tools/staticbuild/README.md

+This folder contains the core script used to build the static library. This README would bring you the information and usages of the script in here. Please be aware, all of the scripts are designed to be run under the root folder.
+
+## `build.sh`
+This script is a wrapper around `build_lib.sh` aimed to simplify the usage of it. It would automatically identify the system version, number of cores and all environment variable settings. Here are the examples you can run this script:


Suggested change

This script is a wrapper around `build_lib.sh` aimed to simplify the usage of it. It would automatically identify the system version, number of cores and all environment variable settings. Here are the examples you can run this script:

This script is a wrapper around `build_lib.sh. It simplifies the things by automatically identifing the system version, number of cores, and all environment variable settings. Here are examples you can run with this script:

aaronmarkham · 2019-01-18T01:13:29Z

tools/staticbuild/README.md

+As the result, users would have a complete static dependencies in `/staticdeps` in the root folder as well as a static-linked `libmxnet.so` file lives in `lib`. You can build your language binding by using the `libmxnet.so`.
+
+## `build_lib.sh`
+This script would clone the most up-to-date master and build the MXNet backend with static library. In order to run that, you should have prepare the the following environment variable:


Suggested change

This script would clone the most up-to-date master and build the MXNet backend with static library. In order to run that, you should have prepare the the following environment variable:

This script clones the most up-to-date master and builds the MXNet backend with a static library. In order to run the static library, you must set the the following environment variables:

aaronmarkham · 2019-01-18T01:14:38Z

tools/staticbuild/README.md

+After running this script, you would have everything you need ready in the `/lib` folder.
+
+## `build_wheel.sh`
+This script is used to build the python package as well as running a sanity test


Suggested change

This script is used to build the python package as well as running a sanity test

This script builds the python package. It also runs a sanity test.

aaronmarkham · 2019-01-18T01:15:41Z

ci/publish/README.md

+- `ubuntu_publish.sh` install all required dependencies for Ubuntu 14.04 for publishing
+- `ubuntu_base.sh` install minimum dependencies required to run the published packages
+
+## Scala publish


Suggested change

## Scala publish

## Scala Publishing

* update scripts and tutorial * add the static test for scala package * kill publish test * fix build issue * address comments

lanking520 requested review from marcoabreu and szha as code owners January 17, 2019 01:51

marcoabreu added the pr-awaiting-review PR is waiting for code review label Jan 17, 2019

zachgk reviewed Jan 17, 2019

View reviewed changes

tools/staticbuild/README.md Outdated Show resolved Hide resolved

marcoabreu reviewed Jan 17, 2019

View reviewed changes

ci/jenkins/Jenkins_steps.groovy Outdated Show resolved Hide resolved

lanking520 added 5 commits January 17, 2019 13:05

update scripts and tutorial

e822d4c

add the static test for scala package

b972bb2

kill publish test

14197b4

fix build issue

8cc7508

address comments

8bb808b

lanking520 force-pushed the static-build-instruction branch from 59a5631 to 8bb808b Compare January 17, 2019 21:05

zachgk approved these changes Jan 18, 2019

View reviewed changes

andrewfayres approved these changes Jan 18, 2019

View reviewed changes

marcoabreu approved these changes Jan 18, 2019

View reviewed changes

marcoabreu merged commit 34a2dc8 into apache:master Jan 18, 2019

aaronmarkham reviewed Jan 18, 2019

View reviewed changes

lanking520 deleted the static-build-instruction branch March 11, 2019 22:28

zachgk pushed a commit to zachgk/incubator-mxnet that referenced this pull request May 16, 2019

Static build instruction for MXNet in general (apache#13914)

619a79a

* update scripts and tutorial * add the static test for scala package * kill publish test * fix build issue * address comments

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Static build instruction for MXNet in general #13914

Static build instruction for MXNet in general #13914

lanking520 commented Jan 17, 2019 •

edited

Loading

pengzhao-intel commented Jan 17, 2019

stu1130 commented Jan 17, 2019

andrewfayres left a comment

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

aaronmarkham Jan 18, 2019

		@@ -0,0 +1,33 @@
		# MXNet Publish Settings

		This folder contains the configuration of restricted node on Jenkins for the publish. It also contains a folder called `scala` that contains everything required for scala publish. In this `README`, we would bring a brief walkthrough of the Jenkins configuration as well as the usages of the scala deployment files.

	This folder contains the configuration of restricted node on Jenkins for the publish. It also contains a folder called `scala` that contains everything required for scala publish. In this `README`, we would bring a brief walkthrough of the Jenkins configuration as well as the usages of the scala deployment files.
	This folder contains the configuration for restricted nodes on Jenkins for the publishing MXNet artifacts. It also contains a folder called `scala` that contains everything required for publishing to Maven. In this `README`, we provide a brief walkthrough of the Jenkins configuration as well as the usage of the Scala deployment files. Python publishing is TBD.


		Apart from that, the script used to create the environment and publish are available under `ci/docker/install`:

		- `ubuntu_publish.sh` install all required dependencies for Ubuntu 14.04 for publishing

	- `ubuntu_publish.sh` install all required dependencies for Ubuntu 14.04 for publishing
	- `ubuntu_publish.sh` installs all required dependencies for Ubuntu 14.04 for publishing

	- `ubuntu_base.sh` install minimum dependencies required to run the published packages
	- `ubuntu_base.sh` installs minimum dependencies required to run the published packages

	Currently Scala publish on Linux is fully supported on jenkins. The `scala/` folder contains all files needed to do the publish. Here is a breif instroduction of the files:
	Currently Scala publish on Linux is fully supported on Jenkins. The `scala/` folder contains all files needed for publishing. Here is a brief introduction of the files:

		@@ -0,0 +1,32 @@
		# MXNet Static build

		This folder contains the core script used to build the static library. This README would bring you the information and usages of the script in here. Please be aware, all of the scripts are designed to be run under the root folder.

	This script is a wrapper around `build_lib.sh` aimed to simplify the usage of it. It would automatically identify the system version, number of cores and all environment variable settings. Here are the examples you can run this script:
	This script is a wrapper around `build_lib.sh. It simplifies the things by automatically identifing the system version, number of cores, and all environment variable settings. Here are examples you can run with this script:

	This script would clone the most up-to-date master and build the MXNet backend with static library. In order to run that, you should have prepare the the following environment variable:
	This script clones the most up-to-date master and builds the MXNet backend with a static library. In order to run the static library, you must set the the following environment variables:

	This script is used to build the python package as well as running a sanity test
	This script builds the python package. It also runs a sanity test.

Static build instruction for MXNet in general #13914

Static build instruction for MXNet in general #13914

Conversation

lanking520 commented Jan 17, 2019 • edited Loading

Description

Checklist

Essentials

pengzhao-intel commented Jan 17, 2019

stu1130 commented Jan 17, 2019

andrewfayres left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

lanking520 commented Jan 17, 2019 •

edited

Loading