Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Refactor]: Refactor data pipeline and support multi-scale training. #311

Merged
merged 6 commits into from
Aug 28, 2021

Conversation

RangiLyu
Copy link
Owner

@RangiLyu RangiLyu commented Aug 22, 2021

Refactor data pipeline

1. Add naive_collate

Replace old collate function with naive_collate. naive_collate only merge meta data dict values into a list from per-image's meta data which load from dataloader workers. Stacking image tensor is moved to the training task.

2. Add stack_batch_img

This function will pad all image tensor into same shape and then stack them into a batch tensor.

3. Support scaling image instead of resizing into a fix shape.

4. Support multi-scale training

Usage: Add multi_scale to data.train

data:
  train:
    name: CocoDataset
    img_path: coco/train2017
    ann_path: coco/annotations/instances_train2017.json
    input_size: [320,320] # [w,h]
    multi_scale: [0.6, 1.4] # scale factor range of multi-scale training

@RangiLyu RangiLyu added enhancement New feature or request refactoring labels Aug 22, 2021
@RangiLyu RangiLyu changed the title [Refactor]: Refactor data pipeline. [Refactor]: Refactor data pipeline and support multi-scale training. Aug 22, 2021
@RangiLyu RangiLyu added the WIP Working in progress. label Aug 22, 2021
@codecov
Copy link

codecov bot commented Aug 22, 2021

Codecov Report

Merging #311 (30923c5) into main (76e025b) will increase coverage by 0.29%.
The diff coverage is 86.33%.

Impacted file tree graph

@@            Coverage Diff             @@
##             main     #311      +/-   ##
==========================================
+ Coverage   73.19%   73.49%   +0.29%     
==========================================
  Files          60       61       +1     
  Lines        3944     4074     +130     
  Branches      649      665      +16     
==========================================
+ Hits         2887     2994     +107     
- Misses        863      881      +18     
- Partials      194      199       +5     
Flag Coverage Δ
unittests 73.49% <86.33%> (+0.29%) ⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

Impacted Files Coverage Δ
nanodet/model/head/gfl_head.py 89.12% <ø> (ø)
nanodet/trainer/dist_trainer.py 28.00% <0.00%> (-1.17%) ⬇️
nanodet/data/collate.py 53.48% <20.00%> (-4.41%) ⬇️
nanodet/trainer/trainer.py 73.26% <55.55%> (-1.19%) ⬇️
nanodet/trainer/task.py 57.41% <63.63%> (+0.47%) ⬆️
nanodet/data/transform/pipeline.py 73.91% <73.33%> (-26.09%) ⬇️
nanodet/data/batch_process.py 95.45% <95.45%> (ø)
nanodet/data/transform/warp.py 85.63% <98.38%> (+4.76%) ⬆️
nanodet/data/dataset/base.py 82.97% <100.00%> (+4.60%) ⬆️
nanodet/data/dataset/coco.py 72.52% <100.00%> (+0.93%) ⬆️
... and 2 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 76e025b...30923c5. Read the comment docs.

@RangiLyu RangiLyu added this to Multi-scale Training in NanoDet V1.0 Plan Aug 27, 2021
@RangiLyu RangiLyu moved this from TODO to In Progress in NanoDet V1.0 Plan Aug 28, 2021
@RangiLyu RangiLyu removed the WIP Working in progress. label Aug 28, 2021
@RangiLyu RangiLyu merged commit d7caaca into main Aug 28, 2021
@RangiLyu RangiLyu moved this from In Progress to Done in NanoDet V1.0 Plan Aug 29, 2021
@RangiLyu RangiLyu deleted the refactor_pipeline branch August 29, 2021 13:07
open-shuaijun pushed a commit to open-shuaijun/nanodet that referenced this pull request Sep 23, 2021
…angiLyu#311)

* [Refactor]: Refactor data pipeline.

* unit test

* unit test

* still update deprecated train and test

* update readme

* fix demo
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request refactoring
Projects
Development

Successfully merging this pull request may close these issues.

None yet

1 participant