Set --rename-dir-limit for gcsfuse to allow dir renames

[romilbhardwaj]

Closes #1281.

The problem was stemming from gcsfuse not allowing directory renames, which is used by tensorflow-dataset. From gcsfuse docs:

Renaming directories is by default not supported. A directory rename cannot be performed atomically in GCS and would therefore be arbitrarily expensive in terms of GCS operations, and for large directories would have high probability of failure, leaving the two directories in an inconsistent state.

However, if your application can tolerate the risks, you may enable renaming directories in a non-atomic way in gcsfuse starting v0.35.0, by setting --rename-dir-limit. If a directory contains fewer files than this limit and no subdirectory, it can be renamed.

This is a tradeoff between (cost, consistency) and functionality - you can have low cost and good consistency but you must disallow renames for large dirs.

SkyPilot now uses --rename-dir-limit 10000, which allows renames for dirs containing up to 10000 files. By setting this limit to 10000, we cap the max cost to $0.05 per dir rename to avoid users accidentally running a big storage bill if they are running frequent renames.

The downside is any dir larger than 10000 files would raise a cryptic Too many open files error like in #1281. I chose 10000 because most common datasets are only a few tens of large files. Lmk if you think we should use a different limit, happy to change.

Tested:

• YAML from Mounted storage failed when downloading tensorflow dataset #1281
• test_smoke::TestStorageWithCredentials

[romilbhardwaj]

On a related note, we need to think about ways to expose these underlying performance vs functionality knobs and flags to power users, perhaps through a ~/.sky/config file. E.g., some users may need --implicit-dirs flag for gcsfuse (mentioned in #1154).

[Michaelvll]

Great catch @romilbhardwaj! I just tested it and it works well for me. Thanks!

[concretevitamin]

we need to think about ways to expose these underlying performance vs functionality knobs and flags to power users, perhaps through a ~/.sky/config file. E.g., some users may need --implicit-dirs flag for gcsfuse (mentioned in #1154).

Why don't we add --implicit-dirs ourselves? A GCP blog that turns on this flag and the flag in this PR by default: https://cloud.google.com/blog/topics/developers-practitioners/cloud-storage-file-system-vertex-ai-workbench-notebooks. Maybe these two flags are enough for ML workloads.