Move ci to balfrin #15
Conversation
cfkanesan
commented
May 16, 2024
cfkanesan
commented
- move ci to balfrin
- update fdb to 5.11.116
| @@ -5,4 +5,4 @@ schema: /scratch/mch/ckanesan/fdb-setup/fdb-schema | |||
| spaces: | |||
| - handler: Default | |||
| roots: | |||
| - path: /scratch/mch/ckanesan/fdb-root/ | |||
| - path: /scratch/mch/ckanesan/fdb_32_39x45_51/ | |||
There was a problem hiding this comment.
Could this fdb root path not also be in /store_new/mch/msopr/icon_workflow_2/?
There was a problem hiding this comment.
Can I just move the files or do I need to archive it again?
There was a problem hiding this comment.
I think you can just move the directory but not 100% sure.
There was a problem hiding this comment.
Seems to be working
tests/test_meteodatalab/conftest.py
Outdated
| if machine == "tsa": | ||
| base_dir = Path("/project/s83c/rz+/icon_data_processing_incubator/") | ||
| elif machine == "balfrin": | ||
| base_dir = Path("/scratch/mch/ckanesan/") |
There was a problem hiding this comment.
Same with this path, could it not also be in /store_new/mch/msopr/icon_workflow_2/?
There was a problem hiding this comment.
Yes, it could but I mentioned during the meeting that I'm trying to figure out a sensible way to reduce the amount of data before moving it. However, I can see that there's a full COSMO-1E run in that folder already. Is there a documented policy regarding storage at CSCS?
There was a problem hiding this comment.
Not that I am aware of. But I find a project folder is more suitable than one of our scratches, then we can all manage/develop the tests.
There was a problem hiding this comment.
Ideally, the testdata would be small enough to be hosted on nexus but that's probably for the future.
There was a problem hiding this comment.
https://confluence.cscs.ch/pages/viewpage.action?pageId=785285325
This was published for store but I don't know if it's also valid for store_new. If it is, the dataset is problematic due to the 50k files per TB rule. I might need to tar it.
