[Feature] Support Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets #2194

jinwonkim93 · 2022-10-16T05:52:53Z

Thanks for your contribution and we appreciate it a lot. The following instructions would make your pull request more healthy and more easily get feedback. If you do not understand some items, don't worry, just make the pull request and seek help from maintainers.

Motivation

Support for Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets

Modification

Please briefly describe what modification is made in this PR.
add dataset
add dataset config
add model config
add data_prepare.md

BC-breaking (Optional)

Does the modification introduce changes that break the backward-compatibility of the downstream repos?
If so, please describe how it breaks the compatibility and how the downstream projects should modify their code to keep compatibility with this PR.

Use cases (Optional)

If this PR introduces a new feature, it is better to list some use cases here, and update the documentation.

Checklist

Pre-commit or other linting tools are used to fix the potential lint issues.
The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
If the modification has potential influence on downstream projects, this PR should be tested with downstream projects, like MMDet or MMDet3D.
The documentation has been modified accordingly, like docstring or example tutorials.

add config file for occlusion face

…3/mmsegmentation into custom/face_occlusion

codecov · 2022-10-17T01:56:08Z

Codecov Report

Base: 88.97% // Head: 88.97% // Increases project coverage by +0.00% 🎉

Coverage data is based on head (70b2853) compared to base (7b09967).
Patch coverage: 81.81% of modified lines in pull request are covered.

Additional details and impacted files

@@           Coverage Diff           @@
##           master    #2194   +/-   ##
=======================================
  Coverage   88.97%   88.97%           
=======================================
  Files         145      146    +1     
  Lines        8735     8746   +11     
  Branches     1473     1474    +1     
=======================================
+ Hits         7772     7782   +10     
- Misses        720      722    +2     
+ Partials      243      242    -1

Flag	Coverage Δ
unittests	`88.97% <81.81%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files	Coverage Δ
mmseg/datasets/face.py	`80.00% <80.00%> (ø)`
mmseg/datasets/__init__.py	`100.00% <100.00%> (ø)`
mmseg/datasets/pipelines/transforms.py	`98.83% <0.00%> (+0.19%)`	⬆️

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

MeowZheng

Many thanks for your contribution, and we are working on do some experiments to test the pr.

xiexinch · 2022-10-17T07:03:56Z

docs/en/dataset_prepare.md

+
+The extracted and upsampled COCO objects images and masks can be found in this [drive](https://drive.google.com/drive/folders/15nZETWlGMdcKY6aHbchRsWkUI42KTNs5?usp=sharing).
+
+Please extract CelebAMask-HQ and 11k Hands images based on the splits found in [drive](https://drive.google.com/drive/folders/15nZETWlGMdcKY6aHbchRsWkUI42KTNs5?usp=sharing). 


I think it's better to provide a script to help other users extract and split these images.

xiexinch · 2022-10-17T07:08:54Z

docs/en/dataset_prepare.md

+## Data Generation
+
+Example script to generate NatOcc dataset 
+
+bash NatOcc.sh
+
+Example script to generate RandOcc dataset
+
+bash RandOcc.sh
+<!-- #endregion -->
+
+```python
+
+```


Is it possible to use codes from the original repo as a reference and redevelop a script?

xiexinch · 2022-10-17T07:12:37Z

configs/_base_/datasets/occlude_face.py

+ img_dir='CelebAMask-HQ-original/image',
+ ann_dir='CelebAMask-HQ-original/mask_edited',
+ split='CelebAMask-HQ-original/split/train.txt',
+ pipeline=train_pipeline)
+
+dataset_train_B = dict(
+ type=dataset_type,
+ data_root=data_root,
+ img_dir='NatOcc-SOT/image',
+ ann_dir='NatOcc-SOT/mask',
+ split='NatOcc-SOT/split/train.txt',
+ pipeline=train_pipeline)
+
+
+dataset_valid = dict(
+ type=dataset_type,
+ data_root=data_root,
+ img_dir='RealOcc/image',
+ ann_dir='RealOcc/mask',
+ split='RealOcc/split/val.txt',
+ pipeline=test_pipeline)
+
+dataset_test = dict(
+ type=dataset_type,
+ data_root=data_root,
+ img_dir='RealOcc/image',
+ ann_dir='RealOcc/mask',
+ split='RealOcc/test.txt',


The structure of the folder is not consistent with what the readme writes, could you also write the directory structure after conversion in the README?

xiexinch · 2022-10-17T07:18:29Z

configs/deeplabv3plus/deeplabv3plus_r101_512x512_C-CM+C-WO-NatOcc-SOT.py

+work_dir = './work_dirs/deeplabv3plus_r101_512x512_C-CM+C-WO-NatOcc-SOT'
+gpu_ids = range(0, 2)


In general, we do not need to set these two configs.

okay i will delete this

xiexinch · 2022-10-17T07:28:22Z

Hi @jinwonkim93,
Many thanks for your contribution, please follow our contribution guide to fix the lint problem.

jinwonkim93 · 2022-10-18T00:05:25Z

Hi @jinwonkim93, Many thanks for your contribution, please follow our contribution guide to fix the lint problem.

fixed lint problem :)
I have re do all the process and it worked well.

xiexinch · 2022-10-18T11:32:44Z

Thanks for updating, we'll do some tests and feedback to you asap :)

xiexinch

Sorry for the late reply, I'm running the generation scripts, after the data generation, we'll test your config then feedback to you.

xiexinch · 2022-11-01T09:48:14Z

docs/en/dataset_prepare.md

+RealOcc.7z
+RealOcc-Wild.7z
+11k-hands_mask.7z
+11k-hands_image.7z


Is it the Hands.zip?

No. the datas can be found in the drive. https://github.com/jinwonkim93/mmsegmentation/blob/c222684c292f1f7edbb40ef761e7ff48a3b73602/docs/en/dataset_prepare.md?plain=1#L403

Perhaps I missed some information, I only found this link to download hand images at https://sites.google.com/view/11khands.
It is Hands.zip but not 11k-hands_image.7z, am I right?

yes. https://sites.google.com/view/11khands visit this site and download Hand images

i will redefine the steps to download materials.

i rewrote the procedure of downloading materials

xiexinch · 2022-11-01T11:09:36Z

docs/en/dataset_prepare.md

+7za x CelebAMask-HQ-masks_corrected.7z -o./CelebAMask-HQ
+#suggest better code if you have
+rsync -a ./CelebAMask-HQ/CelebA-HQ-img/ --files-from=./CelebAMask-HQ-WO-train.txt ./CelebAMask-HQ-WO-Train_img
+basename -s .jpg ./CelebAMask-HQ-train/* > train.txt


Does ./CelebAMask-HQ-train/* correspond to ./CelebAMask-HQ-WO-Train_img?

yes you are right. sorry for the typo error. i will fix it

xiexinch · 2022-11-01T11:13:29Z

docs/en/dataset_prepare.md

+basename -s .jpg ./CelebAMask-HQ-train/* > train.txt
+xargs -n 1 -i echo {}.png < train.txt > mask_train.txt
+rsync -a ./CelebAMask-HQ/CelebAMask-HQ-masks_corrected/ --files-from=./mask_train.txt ./CelebAMask-HQ-WO-Train_mask
+mv train.txt ../data/occlusion-aware-face-dataset


I suggest creating the folder occlusion-aware-face-dataset first.

xiexinch · 2022-11-01T12:14:25Z

docs/en/dataset_prepare.md

+SOURCE_DATASET.MASK_DIR "path/to/mmsegmentation/data_materials/CelebAMask-HQ-WO-Train_mask" \
+OCCLUDER_DATASET.IMG_DIR "path/to/mmsegmentation/data_materials/11k-hands_img" \
+OCCLUDER_DATASET.MASK_DIR "path/to/mmsegmentation/data_materials/11k-hands_masks"


The '/' should be added to the end of the address, otherwise, the mask image will not be found.
Did you meet this problem?

Yes, and i have fix the problem and PR it to the author. Try git pull the latest version

xiexinch · 2022-11-01T12:17:29Z

docs/en/dataset_prepare.md

+├── data
+│ ├── occlusion-aware-face-dataset
+│ │ ├── train.txt
+│ │ ├── NatOcc_hand_sot
+│ │ │ ├── img
+│ │ │ │ ├── {image}.jpg
+│ │ │ ├── mask
+│ │ │ │ ├── {mask}.png
+│ │ ├── NatOcc_object
+│ │ │ ├── img
+│ │ │ │ ├── {image}.jpg
+│ │ │ ├── mask
+│ │ │ │ ├── {mask}.png
+│ │ ├── RandOcc
+│ │ │ ├── img
+│ │ │ │ ├── {image}.jpg
+│ │ │ ├── mask
+│ │ │ │ ├── {mask}.png
+│ │ ├── RealOcc
+│ │ │ ├── img
+│ │ │ │ ├── {image}.jpg
+│ │ │ ├── mask
+│ │ │ │ ├── {mask}.png
+│ │ │ ├── split
+│ │ │ │ ├── val.txt


I think this directory structure should be moved to the end, after the generation scripts.

xiexinch · 2022-11-08T09:30:33Z

Sorry for the late reply, I'm running the generation scripts, after the data generation, we'll test your config then feedback to you.

The data processing script can be run successfully and the experiment can be run normally.

Method	mIoU
DeepLabV3+	95.60

jinwonkim93 · 2022-11-08T10:13:09Z

Sorry for the late reply, I'm running the generation scripts, after the data generation, we'll test your config then feedback to you.

The data processing script can be run successfully and the experiment can be run normally.

Method mIoU
DeepLabV3+ 95.60

oh thank you for testing!

PR fix version to original repository. change to original repository.

jinwonkim93 · 2022-11-11T06:23:23Z

@xiexinch what do you think of merging this PR? Is there more things to do?

xiexinch · 2022-11-11T06:36:35Z

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

jinwonkim93 · 2022-11-11T06:45:03Z

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

There is ablation study on the paper about combination of the dataset. I have done some of the experiments and in my case CelebAMask-HQ-WO with corrected masks (this is original data which does not have occlusion) (C-CM) and one set of hand-occluded face (NatOcc-SOT) and one set of COCO-object occluded dataset (NatOcc) was the best with my additional datasets.

xiexinch · 2022-11-11T07:02:56Z

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

There is ablation study on the paper about combination of the dataset. I have done some of the experiments and in my case CelebAMask-HQ-WO with corrected masks (this is original data which does not have occlusion) (C-CM) and one set of hand-occluded face (NatOcc-SOT) and one set of COCO-object occluded dataset (NatOcc) was the best with my additional datasets.

Could you provide more configs to us? Then we can do the experiments and publish new models.

jinwonkim93 · 2022-11-11T07:08:10Z

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

There is ablation study on the paper about combination of the dataset. I have done some of the experiments and in my case CelebAMask-HQ-WO with corrected masks (this is original data which does not have occlusion) (C-CM) and one set of hand-occluded face (NatOcc-SOT) and one set of COCO-object occluded dataset (NatOcc) was the best with my additional datasets.

Could you provide more configs to us? Then we can do the experiments and publish new models.

sure.

xiexinch · 2022-11-11T07:11:43Z

@xiexinch what do you think of merging this PR? Is there more things to do?

Sorry for the late reply. Did you try to train models with this dataset? The data processing guidelines look good to me, but the training result of DeepLabV3+ is not as good as the official paper, do you have any suggestions?

There is ablation study on the paper about combination of the dataset. I have done some of the experiments and in my case CelebAMask-HQ-WO with corrected masks (this is original data which does not have occlusion) (C-CM) and one set of hand-occluded face (NatOcc-SOT) and one set of COCO-object occluded dataset (NatOcc) was the best with my additional datasets.

Could you provide more configs to us? Then we can do the experiments and publish new models.

sure.

Many thanks for your contribution, this PR is merged :)
You could create a new PR to provide configs to us.

…Segmentation Datasets open-mmlab#2194 * add custom dataset * add face occlusion dataset * add config file for occlusion face * fix format * update prepare.md * formatting * formatting * fix typo error for doc * update downloading process * Update dataset_prepare.md PR fix version to original repository. change to original repository.

…Segmentation Datasets (#2194) add custom dataset add face occlusion dataset add config file for occlusion face fix format update prepare.md formatting formatting fix typo error for doc update downloading process Update dataset_prepare.md PR fix version to original repository. change to original repository.

…Segmentation Datasets (open-mmlab#2194) add custom dataset add face occlusion dataset add config file for occlusion face fix format update prepare.md formatting formatting fix typo error for doc update downloading process Update dataset_prepare.md PR fix version to original repository. change to original repository.

jinwonkim93 and others added 6 commits September 20, 2022 08:47

add custom dataset

e9e196f

Merge branch 'open-mmlab:master' into custom/face_occlusion

d5e79fa

add config file for occlusion face

23c9fd7

Merge pull request #1 from jinwonkim93/face_occlusion

f4022fb

add config file for occlusion face

add face occlusion dataset

e1cc800

Merge branch 'custom/face_occlusion' of https://github.com/jinwonkim9…

33156d0

…3/mmsegmentation into custom/face_occlusion

mm-assistant bot assigned xiexinch Oct 16, 2022

fix format

1ca1780

MeowZheng reviewed Oct 17, 2022

View reviewed changes

MeowZheng requested a review from xiexinch October 17, 2022 03:32

xiexinch reviewed Oct 17, 2022

View reviewed changes

jinwonkim93 and others added 3 commits October 17, 2022 15:30

update prepare.md

69b49cd

formatting

e4a9dd7

formatting

1fc898c

jinwonkim93 added 2 commits October 24, 2022 09:37

Merge branch 'open-mmlab:master' into custom/face_occlusion

f537397

Merge branch 'open-mmlab:master' into custom/face_occlusion

c222684

MeowZheng added this to the 2.0.0rc2 milestone Oct 31, 2022

xiexinch reviewed Nov 1, 2022

View reviewed changes

jinwonkim93 and others added 3 commits November 1, 2022 22:28

Merge branch 'open-mmlab:master' into custom/face_occlusion

280b175

fix typo error for doc

2aaa757

update downloading process

dd59be3

jinwonkim93 added 2 commits November 8, 2022 19:13

Merge branch 'open-mmlab:master' into custom/face_occlusion

62bcc1a

Update dataset_prepare.md

70b2853

PR fix version to original repository. change to original repository.

MeowZheng approved these changes Nov 11, 2022

View reviewed changes

MeowZheng merged commit 6b4c7ff into open-mmlab:master Nov 11, 2022

jinwonkim93 mentioned this pull request Nov 12, 2022

contributing this dataset to mmsegmentation kennyvoo/face-occlusion-generation#7

Closed


		The extracted and upsampled COCO objects images and masks can be found in this [drive](https://drive.google.com/drive/folders/15nZETWlGMdcKY6aHbchRsWkUI42KTNs5?usp=sharing).

		Please extract CelebAMask-HQ and 11k Hands images based on the splits found in [drive](https://drive.google.com/drive/folders/15nZETWlGMdcKY6aHbchRsWkUI42KTNs5?usp=sharing).

		work_dir = './work_dirs/deeplabv3plus_r101_512x512_C-CM+C-WO-NatOcc-SOT'
		gpu_ids = range(0, 2)

[Feature] Support Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets #2194

[Feature] Support Delving into High-Quality Synthetic Face Occlusion Segmentation Datasets #2194

Conversation

jinwonkim93 commented Oct 16, 2022

Motivation

Modification

BC-breaking (Optional)

Use cases (Optional)

Checklist

codecov bot commented Oct 17, 2022 • edited

Codecov Report

MeowZheng left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiexinch commented Oct 17, 2022

jinwonkim93 commented Oct 18, 2022

xiexinch commented Oct 18, 2022

xiexinch left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinwonkim93 Nov 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jinwonkim93 Nov 1, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

xiexinch commented Nov 8, 2022

jinwonkim93 commented Nov 8, 2022

jinwonkim93 commented Nov 11, 2022

xiexinch commented Nov 11, 2022

jinwonkim93 commented Nov 11, 2022 • edited

xiexinch commented Nov 11, 2022

jinwonkim93 commented Nov 11, 2022

xiexinch commented Nov 11, 2022

codecov bot commented Oct 17, 2022 •

edited

jinwonkim93 Nov 1, 2022 •

edited

jinwonkim93 Nov 1, 2022 •

edited

jinwonkim93 commented Nov 11, 2022 •

edited