[feat] Add a new VideoTransforms class #1178
Closed
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
…he existing video transforms
…unit test warning
|
@russellhowes has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@russellhowes has updated the pull request. You must reimport the pull request before landing. |
|
@russellhowes has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Summary: from https://www.internalfb.com/intern/wiki/Pytorch_Ecosystem_Foundation_(EcoF)/PyTorch_Lightning/Operations/Sync_OSS_FBCode/ Reviewed By: ananthsub Differential Revision: D32933988 fbshipit-source-id: 60d9054d7c1f6951910a0892e3001f26930a16f5
…ase for MS loss calculation when no negative pairs found. (facebookresearch#1160) Summary: Pull Request resolved: facebookresearch#1160 Loss value dict needs to be initialized and set to default values when no negatives per-batch (for MS loss calculation). Also, the refiner class should take in encoded layers rather than sequence output. Reviewed By: ebsmothers Differential Revision: D32696676 fbshipit-source-id: b52d8532445141499152353c7893fb83ef6142c4
Summary: Pull Request resolved: facebookresearch#1165 Fix no attribute "cuda" runtime error when testing poolers with cuda. Unit tests pass on github actions as those machines run without gpu. Test Plan: Test unit tests with GPU on learnfair Reviewed By: ebsmothers Differential Revision: D32805045 Pulled By: Ryan-Qiyu-Jiang fbshipit-source-id: 5d2272fc99688d71bbf9cd3b7b71c051deb1de36
Summary: Pull Request resolved: facebookresearch#1149 Add BertImgModel model used in VinVL. The model is a nn.Module based off github.com/microsoft/Oscar/ modified for mmf. This model is usable without MMF trainer. BaseModel class and pretraining to come. Test Plan: ### Unit tests Tests the instantiation and forward pass of the BertImgModel. Used in VinVL classification and pretraining by models in later diffs. Reviewed By: apsdehal Differential Revision: D32574736 Pulled By: Ryan-Qiyu-Jiang fbshipit-source-id: d43e1670cab9c91533d7e6dfc0cdfd7ff7cb59c3
Summary: Pull Request resolved: facebookresearch#1171 Add contrastive head for three-way vinvl loss, which sometimes randomly swaps captions or labels with the task of classifying whether image, text, label triples are correct, or have a swapped text/label. Test Plan: Imported from OSS Reviewed By: apsdehal Differential Revision: D33001604 Pulled By: Ryan-Qiyu-Jiang fbshipit-source-id: b49fa93a18b8b80f16d5fe28963bfa39100ebd0d
…rch#1150) Summary: Pull Request resolved: facebookresearch#1150 Add VinVL classification and pretraining models that use the VinVL BertImgModel trunk. These are nn.Module objects, usable outside of MMF. For example text preprocessing for pretraining, take a look at VinVLTextTokenizer in a later diff. Models forward returns dict with scores and losses. For example usage consult the unit tests or VinVL basemodel. Test Plan: ### Unit Tests Tested forward passes for classification and pretraining models. Pretraining model forward was tested in end-to-end on winoground dataset. Reviewed By: apsdehal Differential Revision: D32574735 Pulled By: Ryan-Qiyu-Jiang fbshipit-source-id: fc1a58db421a33d941b1ddbb5f5a3f35e308e741
Summary: Pull Request resolved: facebookresearch#1151 Add VinVL BaseModel for training and testing from MMF. This model defers to either the classification or pretraining model depending on its config. For an example config consult the project dir or unit tests. Test Plan: ### Unit Tests Tested BaseModel instantiation from config, and forward pass for classification and pretraining. Reviewed By: ebsmothers Differential Revision: D32574738 Pulled By: Ryan-Qiyu-Jiang fbshipit-source-id: 50f8396821effd778c6d5184cd940864fc1eb3b1
Summary: Pull Request resolved: facebookresearch#1167 ViLT, UNITER, Oscar, Scene Graph Benchmark License Test Plan: Imported from OSS Reviewed By: apsdehal Differential Revision: D32851104 Pulled By: Ryan-Qiyu-Jiang fbshipit-source-id: 62a0a9601d48a5f12f3c2d5215e6120c299421b9
Summary: Pull Request resolved: facebookresearch#1170 Send sample list to cuda device when available. Test Plan: Imported from OSS Reviewed By: apsdehal Differential Revision: D33002411 Pulled By: Ryan-Qiyu-Jiang fbshipit-source-id: ce3eccdc67e4ebdc4e3fd84a9d0335eaa32e7bab
…unit test warning
|
@russellhowes has updated the pull request. You must reimport the pull request before landing. |
|
@russellhowes has updated the pull request. You must reimport the pull request before landing. |
|
@russellhowes has updated the pull request. You must reimport the pull request before landing. |
|
@russellhowes has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@russellhowes has updated the pull request. You must reimport the pull request before landing. |
|
@russellhowes has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@russellhowes has updated the pull request. You must reimport the pull request before landing. |
|
@russellhowes has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
@russellhowes has updated the pull request. You must reimport the pull request before landing. |
|
@russellhowes has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Adds a new VideoTransforms class that will apply any transforms from pytorchvideo or torchvision.
All existing processors in video_processors.py can be replaced by one of these transforms.
Replaces references to existing processors in the Charades dataset config.
Once these references are replaced, the original processors can be cleaned up in a follow-up diff.
Adds unit test, fixes a unit test warning in test_processors.py about a collections.abc import
Changes a couple type annotations (mostly for my understanding; these functions may go away soon)