one byte fixing a5000 and a6000 on-prem

[tobi]

on prem machines with a6000 or a5000 didn't work correctly because of a missing comma in the array the grep was for lspci | grep 'A5000A6000' which... isn't a card anyone has.

I think it may make sense to have a look at this script, it feels quite fragile. Probably best would be to do this via nvidia-smi -L and just map against the ./.sky/catalog to find supported gpus.

Also, the way this made things fail was rather frustrating.

Local clusters:
NAME   USER  HEAD_IP       RESOURCES    COMMAND
local  sky   [ip]  [(no GPUs)]  sky launch -c local --  -...

sky status showed (no GPUs) and sky launch fails predictably with AssertionError: (Local(None), 'cloud, region and instance_type must have been set by optimizer')

sky launch -c local totally misreported the error as

Failed to rsync up: /var/folders/wp/6cmt_nw16jj79kd9rj4m777w0000gn/T/sky_local_app_ph50zoz4 -> 
/tmp/sky_local/sky_local_app_ph50zoz4. Ensure that the network is stable, then retry.

I think it's a good idea to detect this failure and report that (remote) ~/.sky/get_resources.py couldn't detect any GPUs or some such thing.

[romilbhardwaj]

Hi @tobi - welcome to the SkyPilot community! Thanks for submitting the PR, lgtm.

Also, thanks for the feedback. I agree, querying nvidia-smi and polling against the catalog will be more robust.

Just wanted to share that onprem support is experimental (and we'll update the docs soon to reflect this, #2024). We are redesigning this feature to be more robust and support a larger set of deployment environments, including multi-tenant clusters. We'd love to learn more about your use-case if you can share more details - I am at [email protected] or Slack. Thanks!

[tobi]

Makes sense. Onprem feels like a key feature to company adoption. We all have existing clusters with low utilization but need to hold on to them because of contracts.

Getting more usage out of them and sharing them better is something that skypilot can help with. That makes the spot and on-demand feature a kind of secondary but powerful feature on top of just really good tooling for making training runs in larger teams.

[romilbhardwaj]

Thanks for sharing @tobi. Running jobs on onprem clusters and spilling them to cloud resources when required is something we want to enable with SkyPilot. We'll keep you posted with our progress.

[tobi]

Thank you.

Really like this project. Great job guys.