[Train] Llama 2 workspace template release tests #37871

kouroshHakha · 2023-07-27T21:34:40Z

Why are these changes needed?

This enables release tests for Llama-2 workspace templates

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
- I've added any new APIs to the API Reference. For example, if I added a
  method in Tune, I've added it in doc/source/tune/api/ under the
  corresponding .rst file.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

…project#37745) Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

This reverts commit 05b0a0f.

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha · 2023-07-28T03:46:06Z

release test passing: https://buildkite.com/ray-project/release-tests-pr/builds/46858#_

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha · 2023-07-28T05:44:27Z

final aws build: https://buildkite.com/ray-project/release-tests-pr/builds/46878

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha · 2023-07-28T06:10:29Z

@justinvyu @can-anyscale This needs your approvals.

aslonnie

(leaving this to @can-anyscale )

can-anyscale

can-anyscale · 2023-07-28T14:26:08Z

doc/source/templates/testing/cluster_envs/04_finetuning_llms_with_deepspeed.yaml

-# See https://hub.docker.com/r/anyscale/ray for full list of
-# available Ray, Python, and CUDA versions.
-base_image: "anyscale/ray:2.6.1-py39-cu117"
+base_image: anyscale/ray:nightly-py39-cu117


cu118? for consistency with byod

can-anyscale · 2023-07-28T14:26:26Z

doc/source/templates/testing/compute_configs/04_finetuning_llms_with_deepspeed/aws_70b_v1.yaml

+ - ResourceType: "instance"
+ Tags:
+ - Key: ttl-hours
+ Value: '24'


nit: new line

can-anyscale · 2023-07-28T14:26:33Z

doc/source/templates/testing/compute_configs/04_finetuning_llms_with_deepspeed/aws_70b_v2.yaml

+ - ResourceType: "instance"
+ Tags:
+ - Key: ttl-hours
+ Value: '24'


nit: new line

can-anyscale · 2023-07-28T14:26:46Z

release/ray_release/buildkite/concurrency.py

@@ -16,7 +16,9 @@
 )

 aws_gpu_cpu_to_concurrency_groups = [
- Condition(min_gpu=9, max_gpu=-1, min_cpu=0, max_cpu=-1, group="large-gpu", limit=4),
+ Condition(
+ min_gpu=9, max_gpu=-1, min_cpu=0, max_cpu=-1, group="large-gpu", limit=100


revert this

can-anyscale · 2023-07-28T14:27:23Z

release/release_tests.yaml

+ working_dir: workspace_templates/04_finetuning_llms_with_deepspeed
+ python: "3.9"
+ frequency: nightly-3x
+ team: train


Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

justinvyu · 2023-07-28T16:07:26Z

release/ray_release/byod/requirements_ml_byod_3.9.in

+bitsandbytes
+wandb
+pytorch-lightning
+protobuf<3.21.0
+torchmetrics
+lm_eval
+tiktoken
+sentencepiece


@can-anyscale So, all release tests are running with the same docker image now? If I want to add a package to one release test, I need to add it to the common byod requirement file? Is there a doc where I can read about this?

Yes. Tho on a PR it would still use your specified cluster_env but on the builds from master it is currently using the byod. @can-anyscale will deprecate the use of cluster envs soon for both PRs and master builds.
BYOD has a couple of benefits:

You don't have to build the envs when launching release tests which means faster time to failure if any

It will make tests more reliable due to consistency of versions. The best case scenario is if we don't pin anything in in the byod requirements files.

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: NripeshN <[email protected]>

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: harborn <[email protected]>

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: e428265 <[email protected]>

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Victor <[email protected]>

kouroshHakha and others added 30 commits July 25, 2023 21:21

[Train] LLM fine-tuning workspace template fix custom resources (ray-…

3da40e9

…project#37745) Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

added llama-2 70b scripts

c259150

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip

cf999c4

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Merge branch 'master' of github.com:ray-project/ray into llama-70b-ft

4d39318

added release tests for 7 and 13B

dc25e03

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

updated README

16ae886

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

updated readme

12f4b37

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip

16be8c1

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

updated scripts

d88f97a

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip

7485d3f

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip

055b065

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

better lamma-7b settings

88cac09

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

1. Fix readme typo, 2. fixed evaluation

32def4f

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

fixed typo in release tests

485168c

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

update readme

9744a13

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

updating cluster_end

4055100

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

fixing release test

cc709dc

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

temp changing concurrency group

159241b

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

test the shell changes

db52ef6

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

added other shells

53d5514

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

reverting activating release tests

05b0a0f

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

reverting concurrency

57f402c

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

lint

4b46723

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

updated docker

c6cef39

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

reverting the random stuff

6929bb8

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

lint

aa673cc

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

update the shell to one

45b59a7

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

code format

f61b019

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

format

b30c4e4

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

Revert "reverting activating release tests"

e7d6c0e

This reverts commit 05b0a0f.

kouroshHakha added 5 commits July 27, 2023 14:38

white space removal

35f76ba

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

testing the path stuff

59b927a

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

byod switching

558213e

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

updated the compiled byod stuff

ae2d09e

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip

a7192bb

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha force-pushed the llama-2-release-test branch from 3cf79d1 to a7192bb Compare July 28, 2023 00:13

kouroshHakha added 3 commits July 27, 2023 17:27

wip

54c70c9

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip

5702c96

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

lint

6100736

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

kouroshHakha added 3 commits July 27, 2023 21:21

wip

6f1a5e6

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip

b61a352

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

wip

49f0b13

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

reverting concurrency

2035342

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

aslonnie reviewed Jul 28, 2023

View reviewed changes

can-anyscale approved these changes Jul 28, 2023

View reviewed changes

kouroshHakha added 2 commits July 28, 2023 08:57

1. cu117->cu118 2. team: train->ml

123b77e

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

lint

fc680c9

Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

justinvyu approved these changes Jul 28, 2023

View reviewed changes

justinvyu reviewed Jul 28, 2023

View reviewed changes

matthewdeng approved these changes Jul 28, 2023

View reviewed changes

kouroshHakha added the tests-ok The tagger certifies test failures are unrelated and assumes personal liability. label Jul 28, 2023

kouroshHakha merged commit 0a079dd into ray-project:master Jul 28, 2023
3 of 26 checks passed

NripeshN pushed a commit to NripeshN/ray that referenced this pull request Aug 15, 2023

[Train] Added Llama 2 workspace template release tests (ray-project#3…

528fbf0

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: NripeshN <[email protected]>

harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023

[Train] Added Llama 2 workspace template release tests (ray-project#3…

e30a5cf

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: harborn <[email protected]>

harborn pushed a commit to harborn/ray that referenced this pull request Aug 17, 2023

[Train] Added Llama 2 workspace template release tests (ray-project#3…

e551109

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]>

arvind-chandra pushed a commit to lmco/ray that referenced this pull request Aug 31, 2023

[Train] Added Llama 2 workspace template release tests (ray-project#3…

8d97d41

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: e428265 <[email protected]>

vymao pushed a commit to vymao/ray that referenced this pull request Oct 11, 2023

[Train] Added Llama 2 workspace template release tests (ray-project#3…

1c2745d

…7871) Signed-off-by: Kourosh Hakhamaneshi <[email protected]> Signed-off-by: Victor <[email protected]>

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Train] Llama 2 workspace template release tests #37871

[Train] Llama 2 workspace template release tests #37871

kouroshHakha commented Jul 27, 2023

kouroshHakha commented Jul 28, 2023

kouroshHakha commented Jul 28, 2023

kouroshHakha commented Jul 28, 2023

aslonnie left a comment

can-anyscale left a comment

can-anyscale Jul 28, 2023

kouroshHakha Jul 28, 2023

can-anyscale Jul 28, 2023

can-anyscale Jul 28, 2023

can-anyscale Jul 28, 2023

kouroshHakha Jul 28, 2023

can-anyscale Jul 28, 2023

kouroshHakha Jul 28, 2023

justinvyu Jul 28, 2023

kouroshHakha Jul 28, 2023

[Train] Llama 2 workspace template release tests #37871

[Train] Llama 2 workspace template release tests #37871

Conversation

kouroshHakha commented Jul 27, 2023

Why are these changes needed?

Related issue number

Checks

kouroshHakha commented Jul 28, 2023

kouroshHakha commented Jul 28, 2023

kouroshHakha commented Jul 28, 2023

aslonnie left a comment

Choose a reason for hiding this comment

can-anyscale left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment