- Reproduce various works based on unified standards 😃
- High-speed training and testing ⚡
- Easy to extend 💭
-
Clone or download this repository.
-
Install the required packages:
pytorch>=1.8.1 wandb>=0.12.10
-
Download the dataset:
The dataset is based on VoxCeleb and is divided into train/valid/test sets according to "Learnable Pins: Crossmodal Embeddings for Person Identity, 2018, ECCV" (901/100/250).
Download
dataset.zip
from Google Drive (2.3GB) and unzip it to the project root directory. The folder structure should be as follows:dataset ├── evals │ ├── test_matching_10.pkl │ ├── test_matching_g.pkl │ ├── test_matching.pkl │ ├── test_retrieval.pkl │ ├── test_verification_g.pkl │ ├── test_verification.pkl │ └── valid_verification.pkl ├── info │ ├── name2gender.pkl │ ├── name2jpgs_wavs.pkl │ ├── name2movies.pkl │ ├── name2voice_id.pkl │ ├── train_valid_test_names.pkl │ └── works │ └── wen_weights.txt ├── face_input.pkl └── voice_input.pkl
-
Learnable Pins: Crossmodal Embeddings for Person Identity, 2018, ECCV
python works/1_pins.py
-
Face-Voice Matching using Cross-modal Embeddings, MM, 2018
python works/2_FV-CME.py
-
On Learning Associations of Faces and Voices, ACCV, 2018
python works/3_LAFV.py
-
Disjoint Mapping Network for Cross-modal Matching of Voices and Faces, ICLR, 2019
python works/11_SS_DIM_VFMR_Barlow.py --name=DIMNet
-
Voice-Face Cross-modal Matching and Retrieval - A Benchmark, 2019
python works/11_SS_DIM_VFMR_Barlow.py --name=VFMR
-
Seeking the Shape of Sound: An Adaptive Framework for Learning Voice-Face Association, CVPR, 2021
python works/5_Wen.py
-
Fusion and Orthogonal Projection for Improved Face-Voice Association, ICASSP, 2022
python works/6_FOP.py
-
Unsupervised Voice-Face Representation Learning by Cross-Modal Prototype Contrast, IJCAI, 2022
python works/7_CMPC.py
-
Self-Lifting: A Novel Framework for Unsupervised Voice-Face Association Learning, ICMR, 2022
python works/9_SL.py
for self-lifting
python works/8_CAE.py
for the CCAE baseline
python works/11_SS_DIM_VFMR_Barlow.py --name=SL-Barlow
for the Barlow Twins baseline
Use wandb to view the training process:
-
Create a
.wb_config.json
file in the project root with the following content:{ "WB_KEY": "Your wandb auth key" }
-
Add
--dryrun=False
to the training command, for example:python main.py --dryrun=False