Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use rapids-build-backend. #1502

Merged
merged 81 commits into from
May 24, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
8ffedf2
Use rapids-build-backend.
bdice Mar 13, 2024
f5e6177
Update dependencies.
bdice Mar 14, 2024
008b20c
Use build module.
bdice Mar 14, 2024
2d4c846
Fix pyproject.toml.
bdice Mar 14, 2024
ab0b17a
Fix key.
bdice Mar 14, 2024
0a2e648
Read GIT_COMMIT.
bdice Mar 14, 2024
697f340
Merge branch 'branch-24.04' into rapids_build_backend
bdice Mar 14, 2024
10ccd85
Install rapids-build-backend.
bdice Mar 14, 2024
f29fa7a
Merge branch 'rapids_build_backend' of github.com:bdice/rmm into rapi…
bdice Mar 14, 2024
952dedc
Fix build-system.requires until packages are available.
bdice Mar 14, 2024
1301e60
Update for changes in rapids-build-backend.
bdice Mar 14, 2024
e4a5abf
Use pip wheel for wheel builds to find it during installation.
bdice Mar 15, 2024
ee87798
Fix scikit-build-core to include pyproject extras.
bdice Mar 15, 2024
8c9c4df
Add dependency on rapids-build-backend.
bdice Mar 15, 2024
747a28c
Remove testing hacks now that packages are available.
bdice Mar 18, 2024
8a28056
Merge remote-tracking branch 'upstream/branch-24.06' into rapids_buil…
bdice Mar 18, 2024
9def158
Update conda recipe.
bdice Mar 18, 2024
ee886d5
Add rapidsai-nightly to dependencies.yaml.
bdice Mar 21, 2024
ba766dd
merge latest branch-24.06
jameslamb May 2, 2024
92949db
fix link in README
jameslamb May 2, 2024
6766df2
add pyproject_dir
jameslamb May 2, 2024
5812808
Merge branch 'rapids_build_backend' of github.com:bdice/rmm into rapi…
jameslamb May 2, 2024
b049e8f
go all the way to the latest dfg
jameslamb May 2, 2024
8a6e4c4
get rapids-build-backend from github
jameslamb May 3, 2024
f0caeac
update scripts
jameslamb May 3, 2024
0a5919f
fix path
jameslamb May 3, 2024
ea2d856
revert changes from local debugging
jameslamb May 3, 2024
5e94960
paths
jameslamb May 3, 2024
b62e232
fix installation in conda build
jameslamb May 3, 2024
a310550
remove duplicate cloning
jameslamb May 3, 2024
0ca672f
revert more stuff from local testing
jameslamb May 3, 2024
4455d9a
merge latest changes
jameslamb May 3, 2024
714f572
thank you rapids-dependency-file-generator, for putting my dependenci…
jameslamb May 3, 2024
9743105
the C++ wheels and Python wheels both need a scikit-build-core dep, b…
jameslamb May 3, 2024
352731c
adjust to presence of cpp wheels
jameslamb May 3, 2024
4554a7e
maybe you need to point at a dependencies.yaml for the suffix-adding …
jameslamb May 3, 2024
793d2ac
Merge branch 'branch-24.06' of github.com:rapidsai/rmm into rapids_bu…
jameslamb May 6, 2024
31fcf91
remove fallback librmm requirement
jameslamb May 6, 2024
535ecab
copyright updates
jameslamb May 6, 2024
7e0988e
add alpha specs
jameslamb May 6, 2024
b212f9b
Merge branch 'branch-24.06' into rapids_build_backend
jameslamb May 7, 2024
eed5625
get better logs, comment out conda jobs
jameslamb May 7, 2024
c291130
get even more logs
jameslamb May 7, 2024
a4c1b4f
use upstream rapids-build-backend
jameslamb May 7, 2024
20c98e8
fix rapids-build-backend installation
jameslamb May 8, 2024
1c78f07
move pin to dependencies.yaml
jameslamb May 8, 2024
9791fef
fix copyrights
jameslamb May 8, 2024
5ade208
fix pin
jameslamb May 8, 2024
f57e124
uncomment more CI
jameslamb May 8, 2024
3be15c6
remove conditions
jameslamb May 8, 2024
09a0b3b
fix rbb installation in conda and devcontainers
jameslamb May 8, 2024
7865d55
Merge branch 'branch-24.06' into rapids_build_backend
jameslamb May 8, 2024
f5eddbb
remove cuda-python rewriting
jameslamb May 8, 2024
2a6b7d5
make GIT_COMMIT stricter
jameslamb May 8, 2024
3190d98
Merge branch 'branch-24.06' of github.com:rapidsai/rmm into rapids_bu…
jameslamb May 9, 2024
b3bb256
make _version.py more generic
jameslamb May 9, 2024
c6d5a78
use rapids-build-backend packages
jameslamb May 9, 2024
620ccd4
copyright
jameslamb May 9, 2024
ae80cb4
fix version specifier for conda
jameslamb May 9, 2024
58f418f
librmm wheel too
jameslamb May 9, 2024
7569d4d
Merge branch 'branch-24.06' into rapids_build_backend
jameslamb May 13, 2024
3c40558
update to latest branch-24.06, move test
jameslamb May 13, 2024
294f7db
revert testing changes
jameslamb May 13, 2024
0494363
Merge branch 'branch-24.06' of github.com:rapidsai/rmm into rapids_bu…
jameslamb May 14, 2024
a507ae1
add ceiling on rapids-build-backend
jameslamb May 14, 2024
dcba4c2
and the conda recipe
jameslamb May 14, 2024
efc5a61
Merge branch 'branch-24.06' of github.com:rapidsai/rmm into rapids_bu…
jameslamb May 16, 2024
dd51662
use rapids-build-backend 0.2.0
jameslamb May 16, 2024
4965a44
more copyright
jameslamb May 16, 2024
92b7133
Revert "more copyright"
jameslamb May 16, 2024
3374f01
Revert "use rapids-build-backend 0.2.0"
jameslamb May 16, 2024
d5d2052
use rapids-build-backend 0.2.0
jameslamb May 16, 2024
8fb02bc
Merge branch 'branch-24.06' into rapids_build_backend
jameslamb May 16, 2024
940d86f
merge branch-24.08
jameslamb May 21, 2024
32f623f
Merge branch 'branch-24.08' into rapids_build_backend
jameslamb May 23, 2024
c6e522c
upgrade to v0.3.0, consolidate rapids-generate-version uses
jameslamb May 23, 2024
cc76e35
rapids-dependency-file-generator version
jameslamb May 23, 2024
b9a31fb
populate __all__ in _version.py
jameslamb May 23, 2024
4aab1c3
set 'rapidsai.disable-cuda=true' in conda builds
jameslamb May 23, 2024
7aeba94
standardize rapids-build-backend version constraint, revert build_cpp…
jameslamb May 24, 2024
1a9b4f1
--config-settings, not --config-setting, for pip
jameslamb May 24, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .pre-commit-config.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ repos:
- id: trailing-whitespace
- id: end-of-file-fixer
- repo: https://github.com/rapidsai/dependency-file-generator
rev: v1.8.0
rev: v1.13.11
hooks:
- id: rapids-dependency-file-generator
args: ["--clean"]
Expand Down
6 changes: 5 additions & 1 deletion build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -170,5 +170,9 @@ fi
# Build and install the rmm Python package
if (( NUMARGS == 0 )) || hasArg rmm; then
echo "building and installing rmm..."
SKBUILD_CMAKE_ARGS="${EXTRA_CMAKE_ARGS}" python -m pip install --no-build-isolation --no-deps ${REPODIR}/python/rmm
SKBUILD_CMAKE_ARGS="${EXTRA_CMAKE_ARGS}" python -m pip install \
--no-build-isolation \
--no-deps \
--config-settings rapidsai.disable-cuda=true \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just pushed one more change... remembered that pip accepts --config-settings (with an s), not --config-setting.

https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-C

Having --config-setting (no s) didn't break anything in CI or even display any warnings, so maybe newer versions of pip have code to capture both. But either way, I think --config-settings (with the s) is the right thing for pip install, given that's what is in that tool's docs.

I'll apply that change to the cudf PR. cc @KyleFromNVIDIA that should be applied to the other rapids-build-backend PRs as well.

${REPODIR}/python/rmm
fi
13 changes: 3 additions & 10 deletions ci/build_python.sh
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
#!/bin/bash
# Copyright (c) 2020-2023, NVIDIA CORPORATION.
# Copyright (c) 2020-2024, NVIDIA CORPORATION.

set -euo pipefail

Expand All @@ -13,20 +13,13 @@ export CMAKE_GENERATOR=Ninja

rapids-print-env

package_name="rmm"
package_dir="python/rmm"

version=$(rapids-generate-version)
commit=$(git rev-parse HEAD)

echo "${version}" > VERSION
sed -i "/^__git_commit__/ s/= .*/= \"${commit}\"/g" "${package_dir}/${package_name}/_version.py"
rapids-generate-version > ./VERSION

rapids-logger "Begin py build"

CPP_CHANNEL=$(rapids-download-conda-from-s3 cpp)

# This calls mambabuild when boa is installed (as is the case in the CI images)
RAPIDS_PACKAGE_VERSION=${version} rapids-conda-retry mambabuild -c "${CPP_CHANNEL}" conda/recipes/rmm
RAPIDS_PACKAGE_VERSION=$(head -1 ./VERSION) rapids-conda-retry mambabuild -c "${CPP_CHANNEL}" conda/recipes/rmm

rapids-upload-conda-to-s3 python
14 changes: 1 addition & 13 deletions ci/build_wheel_cpp.sh
Original file line number Diff line number Diff line change
Expand Up @@ -3,27 +3,15 @@

set -euo pipefail

package_name="librmm"
package_dir="python/librmm"

source rapids-configure-sccache
source rapids-date-string

version=$(rapids-generate-version)
commit=$(git rev-parse HEAD)
rapids-generate-version > ./VERSION

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# This is the version of the suffix with a preceding hyphen. It's used
# everywhere except in the final wheel name.
PACKAGE_CUDA_SUFFIX="-${RAPIDS_PY_CUDA_SUFFIX}"

pyproject_file="${package_dir}/pyproject.toml"

sed -i "s/name = \"${package_name}\"/name = \"${package_name}${PACKAGE_CUDA_SUFFIX}\"/g" ${pyproject_file}
echo "${version}" > VERSION
sed -i "/^__git_commit__/ s/= .*/= \"${commit}\"/g" "${package_dir}/${package_name}/_version.py"

cd "${package_dir}"

python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check
Expand Down
26 changes: 2 additions & 24 deletions ci/build_wheel_python.sh
Original file line number Diff line number Diff line change
Expand Up @@ -9,33 +9,11 @@ package_dir="python/rmm"
source rapids-configure-sccache
source rapids-date-string

version=$(rapids-generate-version)
commit=$(git rev-parse HEAD)

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"

# This is the version of the suffix with a preceding hyphen. It is used
# everywhere except in the final wheel name.
PACKAGE_CUDA_SUFFIX="-${RAPIDS_PY_CUDA_SUFFIX}"

pyproject_file="${package_dir}/pyproject.toml"

sed -i "s/name = \"${package_name}\"/name = \"${package_name}${PACKAGE_CUDA_SUFFIX}\"/g" ${pyproject_file}
echo "${version}" > VERSION
sed -i "/^__git_commit__/ s/= .*/= \"${commit}\"/g" "${package_dir}/${package_name}/_version.py"

alpha_spec=''
if ! rapids-is-release-build; then
alpha_spec=',>=0.0.0a0'
fi

sed -r -i "s/librmm==(.*)\"/librmm${PACKAGE_CUDA_SUFFIX}==\1${alpha_spec}\"/g" ${pyproject_file}
if [[ $PACKAGE_CUDA_SUFFIX == "-cu12" ]]; then
sed -i "s/cuda-python[<=>\.,0-9a]*/cuda-python>=12.0,<13.0a0/g" ${pyproject_file}
fi
rapids-generate-version > ./VERSION

cd "${package_dir}"

RAPIDS_PY_CUDA_SUFFIX="$(rapids-wheel-ctk-name-gen ${RAPIDS_CUDA_VERSION})"
CPP_WHEELHOUSE=$(RAPIDS_PY_WHEEL_NAME="rmm_${RAPIDS_PY_CUDA_SUFFIX}" rapids-download-wheels-from-s3 cpp /tmp/librmm_dist)

python -m pip wheel . -w dist -vvv --no-deps --disable-pip-version-check --find-links "${CPP_WHEELHOUSE}"
Expand Down
4 changes: 3 additions & 1 deletion conda/environments/all_cuda-118_arch-x86_64.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- conda-forge
dependencies:
- breathe>=4.35.0
Expand Down Expand Up @@ -32,7 +33,8 @@ dependencies:
- pytest
- pytest-cov
- python>=3.9,<3.12
- scikit-build-core>=0.7.0
- rapids-build-backend >=0.3.0,<0.4.0.dev0
- scikit-build-core >=0.7.0
- spdlog>=1.12.0,<1.13
- sphinx
- sphinx-copybutton
Expand Down
4 changes: 3 additions & 1 deletion conda/environments/all_cuda-122_arch-x86_64.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,7 @@
# To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.
channels:
- rapidsai
- rapidsai-nightly
- conda-forge
dependencies:
- breathe>=4.35.0
Expand Down Expand Up @@ -31,7 +32,8 @@ dependencies:
- pytest
- pytest-cov
- python>=3.9,<3.12
- scikit-build-core>=0.7.0
- rapids-build-backend >=0.3.0,<0.4.0.dev0
- scikit-build-core >=0.7.0
- spdlog>=1.12.0,<1.13
- sphinx
- sphinx-copybutton
Expand Down
1 change: 1 addition & 0 deletions conda/recipes/rmm/meta.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -62,6 +62,7 @@ requirements:
- cuda-python >=12.0,<13.0a0
{% endif %}
- cython >=3.0.0
- rapids-build-backend >=0.3.0,<0.4.0.dev0
- librmm ={{ version }}
- python
- scikit-build-core >=0.7.0
Expand Down
38 changes: 31 additions & 7 deletions dependencies.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -14,6 +14,7 @@ files:
- develop
- docs
- py_version
- rapids_build_skbuild
- run
- test_python
test_python:
Expand Down Expand Up @@ -43,13 +44,29 @@ files:
pyproject_dir: python/librmm
extras:
table: build-system
includes:
- rapids_build_skbuild
py_cpp_rapids_build:
output: pyproject
pyproject_dir: python/librmm
extras:
table: tool.rapids-build-backend
key: requires
includes:
- build
py_build:
output: pyproject
pyproject_dir: python/rmm
extras:
table: build-system
includes:
- rapids_build_skbuild
py_rapids_build:
output: pyproject
pyproject_dir: python/rmm
extras:
table: tool.rapids-build-backend
key: requires
includes:
- build
- cython_build
Expand All @@ -70,8 +87,19 @@ files:
- test_python
channels:
- rapidsai
- rapidsai-nightly
- conda-forge
dependencies:
rapids_build_skbuild:
common:
- output_types: conda
packages:
- rapids-build-backend >=0.3.0,<0.4.0.dev0
- scikit-build-core >=0.7.0
- output_types: [requirements, pyproject]
packages:
- rapids-build-backend>=0.3.0,<0.4.0.dev0
- scikit-build-core[pyproject]>=0.7.0
build:
common:
- output_types: [conda, requirements, pyproject]
Expand All @@ -83,11 +111,7 @@ dependencies:
- c-compiler
- cxx-compiler
- fmt>=10.1.1,<11
- scikit-build-core>=0.7.0
- spdlog>=1.12.0,<1.13
- output_types: [requirements, pyproject]
packages:
- scikit-build-core[pyproject]>=0.7.0
specific:
- output_types: conda
matrices:
Expand Down Expand Up @@ -137,14 +161,14 @@ dependencies:
- matrix:
cuda: "12.*"
packages:
- librmm-cu12==24.8.*
- librmm-cu12==24.8.*,>=0.0.0a0
- matrix:
cuda: "11.*"
packages:
- librmm-cu11==24.8.*
- librmm-cu11==24.8.*,>=0.0.0a0
- matrix:
packages:
- librmm==24.8.*
- librmm==24.8.*,>=0.0.0a0
checks:
common:
- output_types: [conda, requirements]
Expand Down
7 changes: 6 additions & 1 deletion python/librmm/librmm/_version.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,11 @@
import importlib.resources

__version__ = (
importlib.resources.files("librmm").joinpath("VERSION").read_text().strip()
importlib.resources.files(__package__)
.joinpath("VERSION")
.read_text()
.strip()
)
__git_commit__ = ""

__all__ = ["__git_commit__", "__version__"]
14 changes: 11 additions & 3 deletions python/librmm/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,9 @@
# limitations under the License.

[build-system]
build-backend = "scikit_build_core.build"
build-backend = "rapids_build_backend.build"
requires = [
"cmake>=3.26.4",
"ninja",
"rapids-build-backend>=0.3.0,<0.4.0.dev0",
"scikit-build-core[pyproject]>=0.7.0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.

Expand Down Expand Up @@ -44,6 +43,15 @@ Homepage = "https://github.com/rapidsai/rmm"
[project.entry-points."cmake.prefix"]
librmm = "librmm"

[tool.rapids-build-backend]
build-backend = "scikit_build_core.build"
commit-file = "librmm/GIT_COMMIT"
dependencies-file = "../../dependencies.yaml"
requires = [
"cmake>=3.26.4",
"ninja",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.

[tool.scikit-build]
build-dir = "build/{wheel_tag}"
cmake.build-type = "Release"
Expand Down
20 changes: 14 additions & 6 deletions python/rmm/pyproject.toml
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,9 @@
# limitations under the License.

[build-system]
build-backend = "scikit_build_core.build"
build-backend = "rapids_build_backend.build"
requires = [
"cmake>=3.26.4",
"cuda-python>=11.7.1,<12.0a0",
"cython>=3.0.0",
"librmm==24.8.*",
"ninja",
"rapids-build-backend>=0.3.0,<0.4.0.dev0",
"scikit-build-core[pyproject]>=0.7.0",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.

Expand Down Expand Up @@ -125,6 +121,18 @@ provider = "scikit_build_core.metadata.regex"
input = "rmm/VERSION"
regex = "(?P<value>.*)"

[tool.rapids-build-backend]
build-backend = "scikit_build_core.build"
commit-file = "rmm/GIT_COMMIT"
dependencies-file = "../../dependencies.yaml"
requires = [
"cmake>=3.26.4",
"cuda-python>=11.7.1,<12.0a0",
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point I'd like to consider switching the "default" pyproject.toml to CUDA 12. That can be evaluated later but if we like the idea we could file a build-planning issue.

"cython>=3.0.0",
"librmm==24.8.*,>=0.0.0a0",
"ninja",
] # This list was generated by `rapids-dependency-file-generator`. To make changes, edit ../../dependencies.yaml and run `rapids-dependency-file-generator`.

[tool.pytest.ini_options]
# treat warnings as errors
filterwarnings = [
Expand Down
19 changes: 16 additions & 3 deletions python/rmm/rmm/_version.py
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
# Copyright (c) 2023, NVIDIA CORPORATION.
# Copyright (c) 2023-2024, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
Expand All @@ -15,6 +15,19 @@
import importlib.resources

__version__ = (
importlib.resources.files("rmm").joinpath("VERSION").read_text().strip()
importlib.resources.files(__package__)
.joinpath("VERSION")
.read_text()
.strip()
)
__git_commit__ = ""
try:
__git_commit__ = (
importlib.resources.files(__package__)
.joinpath("GIT_COMMIT")
.read_text()
.strip()
)
except FileNotFoundError:
__git_commit__ = ""

__all__ = ["__git_commit__", "__version__"]
24 changes: 24 additions & 0 deletions python/rmm/rmm/tests/test_version.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Copyright (c) 2024, NVIDIA CORPORATION.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import rmm


def test_version_constants_are_populated():
# __git_commit__ will only be non-empty in a built distribution
assert isinstance(rmm.__git_commit__, str)

# __version__ should always be non-empty
assert isinstance(rmm.__version__, str)
assert len(rmm.__version__) > 0
Loading