[Datasets] Add size parameter to ImageFolderDatasource

[bveeramani]

Why are these changes needed?

If you read a folder with differently-sized images, ImageFolderDatasource errors. This PR fixes the issue by resizing images to a user-specified size.

Related issue number

Checks

• I've run scripts/format.sh to lint the changes in this PR.
• I've included any doc changes needed for https://docs.ray.io/en/master/.
• I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
• Testing Strategy
  • Unit tests
  • Release tests
  • This PR is not tested :(

[c21]

Do we want to do any sanity check of the user-provided size? What's the largest image size we want to support now?

[bveeramani]

Added a check that the size contains positive numbers.

No idea what the large image size should be, if any.

[c21]

LGTM

[jiaodong]

thanks for the quick fix ! One confirmation: i remember you previously mentioned our implementation assumes folder structure is same as imagenet, is this behavior still the same ? Looks like we're making it single root path now.

Also, how would user add / apply labels to each image read with ImageFolderDatasource API ?

[bveeramani]

thanks for the quick fix ! One confirmation: i remember you previously mentioned our implementation assumes folder structure is same as imagenet, is this behavior still the same ? Looks like we're making it single root path now.

Behavior is identical. Previously, create_reader would raise an error if paths contained more than one element.

        if len(paths) != 1:
            raise ValueError(
                "`ImageFolderDatasource` expects 1 path representing the dataset "
                f"root, but it got {len(paths)} paths instead. To fix this "
                "error, pass in a single-element list containing the dataset root "
                '(for example, `paths=["s3:https://imagenet/train"]`)'
            )

To improve UX, I've renamed paths to root and changed the type to str.

Also, how would user add / apply labels to each image read with ImageFolderDatasource API ?

By label, do you mean integer targets used for training? This datasource already creates a "label" column.

[c21]

Looks like there's some test failure in AIR. Could you help fix these? Thanks.

==================== Test output for //doc/source/ray-air/examples:torch_image_batch_pretrained:
--
  | WARNING: The object store is using /tmp instead of /dev/shm because /dev/shm has only 2684354560 bytes available. This will harm performance! You may be able to free up space by deleting files in /dev/shm. If you are inside a Docker container, you can increase /dev/shm size by passing '--shm-size=9.34gb' to 'docker run' (or add it to the run_options list in a Ray cluster config). Make sure to set this to more than 30% of available RAM.
  | Started a local Ray instance. View the dashboard at https://127.0.0.1:8265.
  | Running GPU batch prediction with 1GB data from s3:https://anonymous@air-example-data-2/1G-image-data-synthetic-raw
  | Traceback (most recent call last):
  | File "/root/.cache/bazel/_bazel_root/5fe90af4e7d1ed9fcf52f59e39e126f5/execroot/com_github_ray_project_ray/bazel-out/k8-opt/bin/doc/source/ray-air/examples/torch_image_batch_pretrained.runfiles/com_github_ray_project_ray/doc/source/ray-air/examples/torch_image_batch_pretrained.py", line 33, in <module>
  | dataset = ray.data.read_datasource(ImageFolderDatasource(), paths=[data_url])
  | File "/ray/python/ray/data/read_api.py", line 274, in read_datasource
  | _wrap_and_register_arrow_serialization_workaround(read_args),
  | File "/ray/python/ray/_private/client_mode_hook.py", line 105, in wrapper
  | return func(*args, **kwargs)
  | File "/ray/python/ray/_private/worker.py", line 2239, in get
  | raise value.as_instanceof_cause()
  | ray.exceptions.RayTaskError(TypeError): ray::_get_read_tasks() (pid=3661, ip=172.16.16.3)
  | File "/ray/python/ray/data/read_api.py", line 1136, in _get_read_tasks
  | reader = ds.create_reader(**kwargs)
  | TypeError: create_reader() got an unexpected keyword argument 'paths'
  | ================================================================================
  | (13:28:10) [79 / 100] 14 / 21 tests, 1 failed; Testing //doc/source/ray-air/examples:torch_image_example; 199s local
  | (13:31:32) [80 / 101] 15 / 21 tests, 1 failed; Testing //doc/source/ray-air/examples:torch_incremental_learning; 43s local
  | (13:35:24) [80 / 101] 15 / 21 tests, 1 failed; Testing //doc/source/ray-air/examples:torch_incremental_learning; 275s local
  | (13:39:51) [85 / 106] 20 / 21 tests, 1 failed; Testing //doc/source/ray-air/examples:xgboost_starter; 10s local

[bveeramani]

Looks like there's some test failure in AIR. Could you help fix these? Thanks.

Okay, should be fixed.

[richardliaw]

fix lint?

[jiaodong]

Thanks for updating our examples ! nit: You should be able to remove transforms.Resize(256) now given the resize is already done. Not a blocker.

[richardliaw]

stamp on docs