-
Notifications
You must be signed in to change notification settings - Fork 1
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Move ci to balfrin #15
Changes from 3 commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -30,9 +30,12 @@ def machine(): | |
@pytest.fixture | ||
def data_dir(request, machine): | ||
"""Base data dir.""" | ||
if machine != "tsa": | ||
if machine == "tsa": | ||
base_dir = Path("/project/s83c/rz+/icon_data_processing_incubator/") | ||
elif machine == "balfrin": | ||
base_dir = Path("/scratch/mch/ckanesan/") | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Same with this path, could it not also be in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, it could but I mentioned during the meeting that I'm trying to figure out a sensible way to reduce the amount of data before moving it. However, I can see that there's a full COSMO-1E run in that folder already. Is there a documented policy regarding storage at CSCS? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not that I am aware of. But I find a project folder is more suitable than one of our scratches, then we can all manage/develop the tests. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Ideally, the testdata would be small enough to be hosted on nexus but that's probably for the future. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. https://confluence.cscs.ch/pages/viewpage.action?pageId=785285325 |
||
else: | ||
return None | ||
base_dir = Path("/project/s83c/rz+/icon_data_processing_incubator/") | ||
marker = request.node.get_closest_marker("data") | ||
if marker is None: | ||
return base_dir / "datasets/32_39x45_51" | ||
|
@@ -45,6 +48,8 @@ def data_dir(request, machine): | |
return base_dir / "datasets/32_39x45_51/COSMO-1E_time" | ||
case "reduced-ens": | ||
return base_dir / "datasets/32_39x45_51/COSMO-1E_ens" | ||
case "flexpart": | ||
return base_dir / "data/flexpart" | ||
raise RuntimeError(f"No match for data mark {marker.args[0]}") | ||
|
||
|
||
|
@@ -59,7 +64,7 @@ def fieldextra_path(machine): | |
"""Fieldextra path.""" | ||
conf = { | ||
"tsa": Path("/project/s83c/fieldextra/tsa"), | ||
"balfrin": Path("/users/tsm/proj.aare/fieldextra/v14.3.1/"), | ||
"balfrin": Path("/users/tsm/proj.aare/fieldextra/v14.4.0/"), | ||
} | ||
return conf.get(machine) | ||
|
||
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Could this fdb root path not also be in
/store_new/mch/msopr/icon_workflow_2/
?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can I just move the files or do I need to archive it again?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I think you can just move the directory but not 100% sure.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Seems to be working