Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor DETRs and add C-DETR, DAB-DETR and DINO #9646

Merged
merged 10 commits into from
Jan 28, 2023
Merged

Refactor DETRs and add C-DETR, DAB-DETR and DINO #9646

merged 10 commits into from
Jan 28, 2023

Conversation

jshilong
Copy link
Collaborator

@jshilong jshilong commented Jan 19, 2023 •
edited
Loading

  1. We refactor the Detection Transformer implemetntation(DETR and DeformDETR) in ｍｍdetection.
  2. We add Conditional DETR, DAB-DETR, and DINO in this PR

Details:

  1. Move Transformer from BBoxHead to Detector and design a unified base detector class for DETRs
  2. For the transformer component, the mechanism of register then build is discarded, and it is directly initialized to facilitate code jumping in your IDE.

For more details can refer to the #9534

Compatibility:

  1. The Transformer modules are moved from bbox_head to detector. And the XTransformer classes are discarded. The new detector acts as the original XTransformer. The initialization logic of the original XTransformer should be combined with the _init_layers() and init_weights() of detector. The forward logic should be divided according to DetectionTransformer.forward_transformer() and implemented in pre_transformer(), forward_encoder(), pre_decoder(), and forward_decoder(), respectively.
  2. The new implementation no longer uses the register mechanism to build custom modules, such as encoder, decoder, and position encoding. Hence, the initialization of most modules should be rewritten. The config should also be rewritten to accommodate the new implementation.
  3. XTransformerXcoderLayer modules, including BaseTransformerLayer and DetrTransformerDecoderLayer, were completely re-written, which may have caused BC breakings in downstream codebases. They should implement new modules according to Implement Transformer components and modify their configs.
  4. The file locations of Transformer modules were rearranged. We made a new folder transformer in mmdet/models/layers. The Transformer modules of each DETR have been moved to xxx_layers.py in transformer folder, and other modules in the original transformer.py have been moved to utils.py in transformer folder. They all can be imported from both mmdet/models/layers and mmdet/models/layers/transformer.

Please don't squash commits so we can preserve the great contribution of @Li-Qingyun @LYMDLUT @KeiChiTse

Li-Qingyun and others added 8 commits January 19, 2023 13:14
@Li-Qingyun @jbwang1997
@KeiChiTse @LYMDLUT @LYM-fire @jshilong
[Refactor]: Refactor DETR and Deformable DETR (#8763)
4d30934 
* [Fix] Fix UT to be compatible with pytorch 1.6 (#8707)

* Update

* Update

* force reinstall pycocotools

* Fix build_cuda

* docker install git

* Update

* comment other job to speedup process

* update

* uncomment

* Update

* Update

* Add comments for --force-reinstall

* [Refactor] Refactor anchor head and base head with boxlist (#8625)

* Refactor anchor head

* Update

* Update

* Update

* Add a series of boxes tools

* Fix box type to support n x box_dim boxes

* revert box type changes

* Add docstring

* refactor retina_head

* Update

* Update

* Fix comments

* modify docstring of coder and ioucalculator

* Replace with_boxlist with use_box_type

* fix: fix config of detr-r18

* fix: modified import of MSDeformAttn in PixelDecoder of Mask2Former

* feat: add TransformerDetector as the base detector of DETR-like detectors

* refactor: refactor modules and configs of DETR

* refactor: refactor DETR-related modules in transformer.py

* refactor: refactor DETR-related modules in transformer.py

* fix: add type comments in detr.py

* correct trainloop in detr_r50 config

* fix: modify the parent class of DETRHead to BaseModule

* refactor: refactor modules and configs of Deformable DETR

* fix: modify the usage of num_query

* fix: modify the usage of num_query in configs

* refactor: replace input_proj of detr with ChannelMapper neck

* refactor: delete multi_apply in DETRHead.forward()

* Update detr_r18_8xb2-500e_coco.py

using channel mapper for r18

* change the name of detection_transfomer.py to base_detr.py

* refactor: modify construct binary masks section of forward_pretransformer

* refactor: utilize abstractmethod

* update ABCmeta to make sure reload class TransformerDetector

* some annotation

* some annotation

* some annotation

* refactor: delete _init_transformer in detectors

* refactor: modify args of deformable detr

* refactor: modify about super().__init__()

* Update detr_head.py

Remove the multi feat lvl in function 'predict_by_feat'

* Update detr.py

update init_weights

* some annotation for head

* to make sure the head args the same as detector

* to make sure the head args the same as detector

* some bug

* fix: fix bugs of num_pred in DeformableDETRHead

* add kwargs to transformer

* support MLP and sineembed position

* detele positional encodeing

* delete useless postnorm

* Revert "add kwargs to transformer"

This reverts commit a265c1a.

* Update detr_head.py

Update type and shape of args

* Update detr_head.py

fix args docstring in predict_by_feat

* Update base_detr.py

Update docstring for forward_pretransformer

* Update deformable_detr.py

Fix docstring

* to support conditional detr with reload forward_transformer

* fix: update config files of Two-stage and Box-refine

* replace all bs with batch_size in detr-related files

* update deformable.py and transformer.py

* update docstring in base_detr

* update docstring in base_detr, detr

* doc refine

* Revert "doc refine"

This reverts commit b69da4f.

* doc refine

* doc refine

* updabase_detr, detr, and le layers/transformdoc

* fix doc in base_detr

* add origin repo link

* add origin repo link

* refine doc

* refine doc

* refine doc

* refine doc

* refine doc

* refine doc

* refine doc

* refine doc

* doc: add doc of the first edition of Deformable DETR

* batch_size to bs

* refine doc

* refine doc

* feat: add config comments of specific module

* refactor: refactor base DETR class TransformerDetector

* fix: fix wrong return typehint of forward_encoder in TransformerDetector

* refactor: refactor DETR

* refactor: refactor Deformable DETR

* refactor: refactor forward_encoder and pre_decoder

* fix: fix bugs of new edition

* refactor: small modifications

* fix: move get_reference_points to deformable_encoder

* refactor: merge init_&inter_reference to references in Deformable DETR

* modify docstring of get_valid_ratio in Deformable DETR

* add some docstring

* doc: add docstring of deformable_detr.py

* doc: add docstring of deformable_detr_head.py

* doc: modify docstring of deformable detr

* doc: add docstring of deformable_detr_head.py

* doc: modify docstring of deformable detr

* doc: add docstring of base_detr.py

* doc: refine docstring of base_detr.py

* doc: refine docstring of base_detr.py

* a little change of MLP

* a little change of MLP

* a little change of MLP

* a little change of MLP

* refine config

* refine config

* refine config

* refine doc string for detr

* little refine doc string for detr.py

* tiny modification

* doc: refine docstring of detr.py

* tiny modifications to resolve the conversations

* DETRHead.predict() draft

* tiny modifications to resolve conversations

* refactor: modify arg names and forward strategies of bbox_head

* tiny modifications to resolve the conversations

* support MLP

* fix docsting of function pre_decoder

* fix docsting of function pre_decoder

* fix docstring

* modifications for resolving conversations

* refactor: eradicate key_padding_mask args

* refactor: eradicate key_padding_mask args

* fix: fix bug of deformable detr and resolve some conversations

* refactor: rename base class with DetectionTransformer and other modifications

* fix: fix config of detr

* fix the bug of init

* fix: fix init_weight of DETR and Deformable DETR

* resolve conflict

* fix auto-merge bug

* fix pre-commit bug

* refactor: move the position of encoder and decoder

* delete Transformer in ci test

* delete Transformer in ci test

Co-authored-by: jbwang1997 <[email protected]>
Co-authored-by: KeiChiTse <[email protected]>
Co-authored-by: LYMDLUT <[email protected]>
Co-authored-by: lym <[email protected]>
Co-authored-by: Kei-Chi Tse <[email protected]>
@LYMDLUT @jshilong
Add unitests for detr 3.x (#9089)
5594cb5
@KeiChiTse @jshilong
[Refactor] Change to BNC data flow for all DETRs (#9460)
ea8e30d 
* fix data format from nbc to bnc in detr and deformable-detr

* set 'batch_first' to True in deformable attention
Co-authored-by: QingyunLi <[email protected]>
@LYMDLUT @jshilong
Refactor detr 3.x conditional detr (#9405)
7d3e0f0 
* Add conditional detr to 3.0
Co-authored-by: lym <[email protected]>
@Li-Qingyun @KeiChiTse @jshilong
[Feat] Add DINO on MMDetection 3.x (#9149)
788d6f3 
* feat: add config of DINO_4s_R50_50e

* feat: add detector file of DINO

* feat: add head file of DINO

* fix: align the inference results

* fix: align the loss

* refactor: move decoder to layers

* small modifications to resolve recent conversations

* refactor: rewrite device manner and other small modifications

* refactor: refactor multiple mode of pre_decoder

* doc: complementary the docstring of dino.py

* refactor & doc: refactor DINOHead and add some docstring of the dino_head.py

* fix & doc: fix bugs and add docs of dino_transformer.py

* refactor: rewrite CdnQueryGenerator

* refactor: refactor CdnQueryGenerator.

* refactor: modify cdn

* refactor: rewrite and add docstring to CdnQueryGenerator.generate_dn_bbox_query

* doc: fix return format

* doc: add docstring of CdnQueryGenerator

* feat: add 91-cls configs of detr and deformable detr

* refactor: rewrite generate_dn_mask

* refactor: rewrite collate_dn_queries

* fix & doc: fix bug of get_dn_target and add some docstring

* doc: write docstring of DINOHead

* feat: add configs of R50-24e and R50-36e

* fix: align loss

* fix: delete original __call__ of CdnQueryGenerator

* fix: add init of query_embedding

* fix: fix typo of dino config

* fix: move convert_coordinate_to_encoding and modify num_query->num_queries num_key->num_keys

* resolve the conversations

* delete some TODO

* add yapf diasble to layers/__init__.py

* fix unit test

* commit modification in pr of DINO

* fix data format from nbc to bnc in detr and deformable-detr

* fix 'gen_encoder_output_proposals' for two-stage deformable-detr

* fix 'gen_encoder_output_proposals' for two-stage deformable-detr

* set 'batch_first' to True in deformable attention

* fix error

* fix ut

* add assert for batch_first

* remove all 91cls-related codes

* doc: add loss_by_feat comment

* modify pre_decoder of DeformableDETR

* delete useless comments

* refactor: change to BNC data flow for DINO

* ut: add unit test of DINO

* add README.md of dino

* fix wrong box_noise_scale doc

* fix __init__.py of layer

Co-authored-by: KeiChiTse <[email protected]>
@KeiChiTse @Li-Qingyun @jshilong
[Refactor]: Refactor DAB-DETR in MMDetection 3.x (#9252)
a44b504 
* resolve refactor conflict w.o. pre-commit hooks

* fixed error and finished alignment

* supprot 91 cls and remove batch_first

* delete iter_update, keep return intermediate

* delete posHW

* substitute 'num_query' to 'num_queries'

* change 'gen_sineembed_for_position' to 'convert_coordinate_to_encoding'

* resolve extra comments

* fix error

* fix error

* fix data path

* support 91 cls temporarily

* resolve extra comments

* fix num_keys, num_feats

* delete reg_branches in decoder_inputs_dict

* fix docstring

* fix docstring

* commit modification in pr of DINO

* fix data format from nbc to bnc in detr and deformable-detr

* fix 'gen_encoder_output_proposals' for two-stage deformable-detr

* fix 'gen_encoder_output_proposals' for two-stage deformable-detr

* set 'batch_first' to True in deformable attention

* fix error

* fix ut

* add assert for batch_first

* remove 91 cls

* modify pre_decoder of DeformableDETR

* delete useless comments

* bnc data flow w.o. merge detr and def-detr

* assert batch first flag in conditional attention, fix error

* add unit test for dab-detr

* fix doc

* disable yapf hook

* move conditional attention to trm/layers

* fix name and add doc

* fix doc

* add loss_and_predict for head

* fix doc and typehint

* fix doc and typehint

* modify batch first assert for attention

* change Dab to DAB

* rename file and function

* make dab-detr head inherit conditional detr head

* fix doc

* fix doc

Co-authored-by: QingyunLi <[email protected]>
@Li-Qingyun @KeiChiTse @jshilong
[Fix] Fix MaskFormer and Mask2Former (#9515)
fbcb8a9 
Co-authored-by: Kei-Chi Tse <[email protected]>
@Li-Qingyun @jshilong
[Fix] Update links in readme of DETRs and MaskFormers (#9634)
4c11117
@jshilong
Copy link
Collaborator Author

I will fix the CI asap

Li-Qingyun
Li-Qingyun previously approved these changes Jan 19, 2023
View reviewed changes
Copy link
Contributor

@Li-Qingyun Li-Qingyun left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have checked the files.
The issue of unit test requires a little discussion.

tests/test_models/test_layers/test_plugins.py Outdated Show resolved Hide resolved
@ZwwWayne ZwwWayne merged commit 1a264de into dev-3.x Jan 28, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Li-Qingyun Li-Qingyun Li-Qingyun left review comments

Assignees

@RangiLyu RangiLyu

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
6 participants
@jshilong @Li-Qingyun @RangiLyu @ZwwWayne @LYMDLUT @KeiChiTse