[CLIP][IMAGE TRANSFORMS] Image transforms for clip encoder

[felipemello1]

Context

• add a new feature
• fix a bug
• update tests and/or documentation
• other (please add here)

Added the image transforms for clip.

Builder with the model specific values in:
torchtune/models/clip/_model_builders.py

Clip specific use of transforms in:
torchtune/models/clip/_transforms.py

Vision transforms in:
torchtune/modules/transforms/vision

Every transform is a function.

Algorithm TLDR:

1. Gets list of possible resolutions based on tile_size and max_num_tiles;
2. Finds resolutions that best fits the image;
3. Resizes with distortion
4. Pads
5. Normalizes
6. tile_crop -> output is of shape [num_tiles, 3, tile_size, tile_size]

Changelog

Added torchvision to the requirements

Test plan

Every function is covered, some indirectly. All tests pass.

• run pre-commit hooks and linters (make sure you've first installed via pre-commit install)
• add unit tests for any new functionality
• update docstrings for any new or updated methods or classes
• run unit tests via pytest tests
• run recipe tests via pytest tests -m integration_test
• manually run any new or modified recipes with sufficient proof of correctness
  • include relevant commands and any other artifacts in this summary (pastes of loss curves, eval results, etc.)

docs

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/torchtune/1084

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[pbontrager]

This is a great first pass over adding all the functionality needed for high resolution CLIP image transforms. I've left a lot of comments on the structure of the transforms as you're adding the first ones to the library. After we update these I'll take a pass over the tests and docs.

[joecummings]

Nit: These examples are not rendering correctly in docs.

[RdoubleA]

I think you need to add newlines between Args, Returns, Examples
Or you need to add a colon

Suggested change

	Examples:
	Examples::

[joecummings]

nit: These examples are not rendering correctly in docs.

[RdoubleA]

Looks awesome overall, thanks for all the work you put into this. One overall question I had is the distinction between the CLIP transform which lives in torchtune/models and the other transforms in modules/transforms/vision. Will only the model transforms be classes with __call__ and all the transforms that live in modules/transforms/ be normal functions?

Also, what is the expected approach for a user to make a new model transform? I had originally thought there would be common image transforms that they can just Compose together in the model builder, but I see that CLIPImageTransform is a bit more involved than just piping transforms. So is this the overall user flow we want to enforce:

• all general transforms are standalone functions in modules
• to make a new model transform, user needs to define a class that uses the general transform functions in any way they want
• then they create a builder function that parametrizes the class and can be instantiated in the config

[RdoubleA]

is there a paper ref for these numbers that we can add to the docstring?

[RdoubleA]

I think you need to add newlines between Args, Returns, Examples
Or you need to add a colon

Suggested change

	Examples:
	Examples::

[RdoubleA]

or maybe these lines in between the code need to be tabbed back

[RdoubleA]

No major concerns on my end - thanks for pushing this through. You might have to wait until recipe tests is fixed on main before landing this.

Also curious if we plan to add model transforms to the api_ref_models.rst, cc @pbontrager . But this can be a follow-up

[pbontrager]

Everything looks good here. I'll approve this, but I think you should update the check for possible_resolutions to explicitly check for None.

[pbontrager]

You have to explicitly check for None, otherwise 0 or empty tuples resolve as false

[felipemello1]

If possible_resolutions is None or is empty or anything that is not a list with items, we must activate this condition to find possible_resolutions

[felipemello1]

[pbontrager]

This else doesn't make sense to me

[lucylq]

if target_size < image_size, it will crop the image.

From _get_max_res_without_distortion, it seems like we always resize such that the image remains within bounds of target_size - is cropping still required?

[lucylq]

Could this be simplified to:

new_width = min(math.floor(original_width * scale_h), target_width)
new_height = min(math.floor(original_height * scale_w), target_height)