[Datasets] Improve size estimation of image folder data source #27219

c21 · 2022-07-28T21:04:27Z

Signed-off-by: Cheng Su [email protected]

Why are these changes needed?

This PR is to improve in-memory data size estimation of image folder data source. Before this PR, we use on-disk file size as estimation of in-memory data size of image folder data source. This can be inaccurate due to image compression and in-memory image resizing.

Given size and mode is set to be optional in #27295, so change this PR to tackle the simple case when size and mode are both provided.

size and mode is provided: just calculate the in-memory size based on the dimensions, not need to read any image (this PR)
size or mode is not provided: need sampling to determine the in-memory size (will do in another followup PR).

Here is example of estiamted size for our test image folder

>>> import ray
>>> from ray.data.datasource.image_folder_datasource import ImageFolderDatasource
>>> root = "example:https://image-folders/different-sizes"
>>> ds = ray.data.read_datasource(ImageFolderDatasource(), root=root, size=(64, 64), mode="RGB")
>>> ds.size_bytes()
40310
>>> ds.fully_executed().size_bytes()
37428

Without this PR:

>>> ds.size_bytes()
18978

Related issue number

Checks

I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
I've run scripts/format.sh to lint the changes in this PR.
I've included any doc changes needed for https://docs.ray.io/en/master/.
I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
Testing Strategy
- Unit tests
- Release tests
- This PR is not tested :(

c21 · 2022-07-28T21:06:19Z

The added unit test of this PR depends on #27156. But this PR itself is also ready for review.

ericl · 2022-07-28T21:10:53Z

python/ray/data/datasource/image_folder_datasource.py

+ non_empty_path_and_size = list(
+ filter(lambda p: p[1] > 0, zip(self._paths, self._file_sizes))
+ )
+ assert len(non_empty_path_and_size) > 0


Should we raise ValueError or return the default estimate in this case?

@ericl - sure, return default value as 1 (to use on-disk file size) in this case. We anyway will throw exception in other places when input data source is empty.

python/ray/data/datasource/image_folder_datasource.py

ericl

So actually, since size is required, can't we statically compute the image size based on the resize dimensions? WHdtype right? No need to perform a disk IO.

c21 · 2022-07-29T00:08:32Z

So actually, since size is required, can't we statically compute the image size based on the resize dimensions? WHdtype right? No need to perform a disk IO.

@ericl - that's what I was planning to do in first place. But with size, we only know the width and height. We don't know the 3rd dimension - whether it's a RGB or grayscale image, and we have to assume each element in image matrix is a uint8 or something.

clarkzinzow · 2022-07-29T00:09:21Z

python/ray/data/datasource/image_folder_datasource.py

+ self._reader_args.get("root"),
+ self._reader_args.get("size"),
+ )
+ single_image_memory_size = single_image_block.memory_usage().sum()


Don't we need deep=True here? https://pandas.pydata.org/docs/reference/api/pandas.DataFrame.memory_usage.html

@clarkzinzow - good catch, I think we should, let me add it.

@clarkzinzow - added.

clarkzinzow · 2022-07-29T00:14:22Z

@c21 We at least know that the data will always be uint8 bytes:

ray/python/ray/data/datasource/image_folder_datasource.py

Line 143 in 4d61432

image = skimage.util.img_as_ubyte(image)

As for the channel dimension, can't you just do 8 * np.prod(size), and that should work generically for any image shapes, e.g. (H, W, C)?

ericl

Got it, I guess this seems like a reasonable approach if we have to decode the image anyway.

ericl · 2022-07-29T00:15:19Z

/tmp/artifacts/.failed_test_logs/python::ray::data::tests::test_dataset_formats.py::test_image_folder_reader_estimate_data_size[64-30000-5]_1659048762.8525.zip

clarkzinzow · 2022-07-29T00:17:03Z

Ah it looks like it's valid for the user to only pass (H, W) while skimage will preserve the channels... That's unfortunate. https://scikit-image.org/docs/dev/api/skimage.transform.html#skimage.transform.resize

c21 · 2022-07-29T00:18:08Z

As for the channel dimension, can't you just do 8 * np.prod(size), and that should work generically for any image shapes, e.g. (H, W, C)?

@clarkzinzow - I may miss something, but how do you know the 3rd dimension of image - whether it's a RGB (h, w, 3)or grayscale (h, w, 1)?

c21 · 2022-07-29T00:19:51Z

@ericl - yeah the unit test is failing now, because this PR depends on #27156 (#27219 (comment)). I tested locally it passed when I stacked this PR on top of the other one. But github does not allow publishing stacked PRs.

python/ray/data/datasource/image_folder_datasource.py

c21 · 2022-07-29T19:12:51Z

Hold off on this, as more discussion around image resizing is still on-going. cc @bveeramani, @clarkzinzow and @amogkam

c21 · 2022-08-10T22:50:01Z

Given size and mode is set to be optional in #27295, so change this PR to tackle the simple case when size and mode are both provided.

size and mode is provided: just calculate the in-memory size based on the dimensions, not need to read any image (this PR)
size or mode is not provided: need sampling to determine the in-memory size (will do in another followup PR).

c21 · 2022-08-10T22:51:09Z

@ericl, @jianoaix, @clarkzinzow - could you help take a look when you have time? Thanks.

jianoaix · 2022-08-11T23:14:01Z

python/ray/data/datasource/image_folder_datasource.py

+
+ sampling_duration = time.perf_counter() - start_time
+ if sampling_duration > 5:
+ logger.info(


Should this be a warning message if this is considered a bit too long?

@jianoaix - make sense, updated.

Signed-off-by: Cheng Su <[email protected]>

…roject#27219) This PR is to improve in-memory data size estimation of image folder data source. Before this PR, we use on-disk file size as estimation of in-memory data size of image folder data source. This can be inaccurate due to image compression and in-memory image resizing. Given `size` and `mode` is set to be optional in ray-project#27295, so change this PR to tackle the simple case when `size` and `mode` are both provided. * `size` and `mode` is provided: just calculate the in-memory size based on the dimensions, not need to read any image (this PR) * `size` or `mode` is not provided: need sampling to determine the in-memory size (will do in another followup PR). Here is example of estiamted size for our test image folder ``` >>> import ray >>> from ray.data.datasource.image_folder_datasource import ImageFolderDatasource >>> root = "example:https://image-folders/different-sizes" >>> ds = ray.data.read_datasource(ImageFolderDatasource(), root=root, size=(64, 64), mode="RGB") >>> ds.size_bytes() 40310 >>> ds.fully_executed().size_bytes() 37428 ``` Without this PR: ``` >>> ds.size_bytes() 18978 ```

…roject#27219) This PR is to improve in-memory data size estimation of image folder data source. Before this PR, we use on-disk file size as estimation of in-memory data size of image folder data source. This can be inaccurate due to image compression and in-memory image resizing. Given `size` and `mode` is set to be optional in ray-project#27295, so change this PR to tackle the simple case when `size` and `mode` are both provided. * `size` and `mode` is provided: just calculate the in-memory size based on the dimensions, not need to read any image (this PR) * `size` or `mode` is not provided: need sampling to determine the in-memory size (will do in another followup PR). Here is example of estiamted size for our test image folder ``` >>> import ray >>> from ray.data.datasource.image_folder_datasource import ImageFolderDatasource >>> root = "example:https://image-folders/different-sizes" >>> ds = ray.data.read_datasource(ImageFolderDatasource(), root=root, size=(64, 64), mode="RGB") >>> ds.size_bytes() 40310 >>> ds.fully_executed().size_bytes() 37428 ``` Without this PR: ``` >>> ds.size_bytes() 18978 ``` Signed-off-by: Stefan van der Kleij <[email protected]>

…roject#27219) This PR is to improve in-memory data size estimation of image folder data source. Before this PR, we use on-disk file size as estimation of in-memory data size of image folder data source. This can be inaccurate due to image compression and in-memory image resizing. Given `size` and `mode` is set to be optional in ray-project#27295, so change this PR to tackle the simple case when `size` and `mode` are both provided. * `size` and `mode` is provided: just calculate the in-memory size based on the dimensions, not need to read any image (this PR) * `size` or `mode` is not provided: need sampling to determine the in-memory size (will do in another followup PR). Here is example of estiamted size for our test image folder ``` >>> import ray >>> from ray.data.datasource.image_folder_datasource import ImageFolderDatasource >>> root = "example:https://image-folders/different-sizes" >>> ds = ray.data.read_datasource(ImageFolderDatasource(), root=root, size=(64, 64), mode="RGB") >>> ds.size_bytes() 40310 >>> ds.fully_executed().size_bytes() 37428 ``` Without this PR: ``` >>> ds.size_bytes() 18978 ```

…roject#27219) This PR is to improve in-memory data size estimation of image folder data source. Before this PR, we use on-disk file size as estimation of in-memory data size of image folder data source. This can be inaccurate due to image compression and in-memory image resizing. Given `size` and `mode` is set to be optional in ray-project#27295, so change this PR to tackle the simple case when `size` and `mode` are both provided. * `size` and `mode` is provided: just calculate the in-memory size based on the dimensions, not need to read any image (this PR) * `size` or `mode` is not provided: need sampling to determine the in-memory size (will do in another followup PR). Here is example of estiamted size for our test image folder ``` >>> import ray >>> from ray.data.datasource.image_folder_datasource import ImageFolderDatasource >>> root = "example:https://image-folders/different-sizes" >>> ds = ray.data.read_datasource(ImageFolderDatasource(), root=root, size=(64, 64), mode="RGB") >>> ds.size_bytes() 40310 >>> ds.fully_executed().size_bytes() 37428 ``` Without this PR: ``` >>> ds.size_bytes() 18978 ``` Signed-off-by: Artur Niederfahrenhorst <[email protected]>

c21 requested review from ericl, scv119, clarkzinzow, jjyao and jianoaix as code owners July 28, 2022 21:04

c21 assigned ericl, clarkzinzow and jianoaix Jul 28, 2022

ericl reviewed Jul 28, 2022

View reviewed changes

ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jul 28, 2022

c21 commented Jul 28, 2022

View reviewed changes

python/ray/data/datasource/image_folder_datasource.py Outdated Show resolved Hide resolved

c21 force-pushed the image branch from 7f74e68 to 2bd46f5 Compare July 28, 2022 21:51

c21 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jul 28, 2022

ericl reviewed Jul 29, 2022

View reviewed changes

ericl added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jul 29, 2022

clarkzinzow reviewed Jul 29, 2022

View reviewed changes

ericl approved these changes Jul 29, 2022

View reviewed changes

jianoaix reviewed Jul 29, 2022

View reviewed changes

python/ray/data/datasource/image_folder_datasource.py Show resolved Hide resolved

c21 force-pushed the image branch from 4d61432 to f503784 Compare July 29, 2022 01:05

c21 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jul 29, 2022

jianoaix approved these changes Jul 29, 2022

View reviewed changes

c21 added the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Jul 29, 2022

c21 force-pushed the image branch from f503784 to aaeaeca Compare August 10, 2022 22:46

c21 removed the @author-action-required The PR author is responsible for the next step. Remove tag to send back to the reviewer. label Aug 10, 2022

jianoaix approved these changes Aug 11, 2022

View reviewed changes

c21 added 6 commits August 11, 2022 17:07

Improve image folder data source size estimation

641752e

Signed-off-by: Cheng Su <[email protected]>

Address comment to use default value when all image files are empty

eecd04a

Signed-off-by: Cheng Su <[email protected]>

Add logger and reformat

de9890d

Signed-off-by: Cheng Su <[email protected]>

Address comments and rebase to latest master

302e230

Signed-off-by: Cheng Su <[email protected]>

Take mode into consideration

83adca3

Signed-off-by: Cheng Su <[email protected]>

Change log level

f2300ff

Signed-off-by: Cheng Su <[email protected]>

c21 force-pushed the image branch from aaeaeca to f2300ff Compare August 12, 2022 00:07

ericl merged commit 7c7828f into ray-project:master Aug 12, 2022

c21 deleted the image branch August 12, 2022 22:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Datasets] Improve size estimation of image folder data source #27219

[Datasets] Improve size estimation of image folder data source #27219

c21 commented Jul 28, 2022 •

edited

Loading

c21 commented Jul 28, 2022

ericl Jul 28, 2022

c21 Jul 28, 2022

ericl left a comment

c21 commented Jul 29, 2022

clarkzinzow Jul 29, 2022

c21 Jul 29, 2022

c21 Jul 29, 2022

clarkzinzow commented Jul 29, 2022 •

edited

Loading

ericl left a comment

ericl commented Jul 29, 2022

clarkzinzow commented Jul 29, 2022

c21 commented Jul 29, 2022

c21 commented Jul 29, 2022

c21 commented Jul 29, 2022

c21 commented Aug 10, 2022

c21 commented Aug 10, 2022

jianoaix Aug 11, 2022

c21 Aug 12, 2022

[Datasets] Improve size estimation of image folder data source #27219

[Datasets] Improve size estimation of image folder data source #27219

Conversation

c21 commented Jul 28, 2022 • edited Loading

Why are these changes needed?

Related issue number

Checks

c21 commented Jul 28, 2022

ericl Jul 28, 2022

Choose a reason for hiding this comment

c21 Jul 28, 2022

Choose a reason for hiding this comment

ericl left a comment

Choose a reason for hiding this comment

c21 commented Jul 29, 2022

clarkzinzow Jul 29, 2022

Choose a reason for hiding this comment

c21 Jul 29, 2022

Choose a reason for hiding this comment

c21 Jul 29, 2022

Choose a reason for hiding this comment

clarkzinzow commented Jul 29, 2022 • edited Loading

ericl left a comment

Choose a reason for hiding this comment

ericl commented Jul 29, 2022

clarkzinzow commented Jul 29, 2022

c21 commented Jul 29, 2022

c21 commented Jul 29, 2022

c21 commented Jul 29, 2022

c21 commented Aug 10, 2022

c21 commented Aug 10, 2022

jianoaix Aug 11, 2022

Choose a reason for hiding this comment

c21 Aug 12, 2022

Choose a reason for hiding this comment

c21 commented Jul 28, 2022 •

edited

Loading

clarkzinzow commented Jul 29, 2022 •

edited

Loading