Skip to content
/ IDT Public

Official repository for the paper "Image Deraining Transformer".

License

Notifications You must be signed in to change notification settings

jiexiaou/IDT

Repository files navigation

Jie Xiao, Xueyang Fu, Aiping Liu, Feng Wu, Zheng-Jun Zha


Update:


Abstract: Existing deep learning based de-raining approaches have resorted to the convolutional architectures. However, the intrinsic limitations of convolution, including local receptive fields and independence of input content, hinder the model's ability to capture long-range and complicated rainy artifacts. To overcome these limitations, we propose an effective and efficient transformer-based architecture for the image de-raining. Firstly, we introduce general priors of vision tasks, i.e., locality and hierarchy, into the network architecture so that our model can achieve excellent de-raining performance without costly pre-training. Secondly, since the geometric appearance of rainy artifacts is complicated and of significant variance in space, it is essential for de-raining models to extract both local and non-local features. Therefore, we design the complementary window-based transformer and spatial transformer to enhance locality while capturing long-range dependencies. Besides, to compensate for the positional blindness of self-attention, we establish a separate representative space for modeling positional relationship, and design a new relative position enhanced multi-head self-attention. In this way, our model enjoys powerful abilities to capture dependencies from both content and position, so as to achieve better image content recovery while removing rainy artifacts. Experiments substantiate that our approach attains more appealing results than state-of-the-art methods quantitatively and qualitatively.

Method

IDT

Details

Dataset

For full-size image

To evaluate the image with arbitrary size, we first split the image to overlapped $128\times 128$ patches, and merge evaluated patches back to original resolution. Compared with directly averaging the overlapped zone, our testing precedure helps to mitigate block artifacts. Please see test_full_size.py for implementation. Merge

Demo

To test the pre-trained IDT model on full size images:

python test_full_size.py --arch IDT --save_path your_save_path --data_path your_input_path --weights pretrained_weights

Results

For research convinience, we release both derained patches as well as full-size images.

Evaluation

Full-size Evaluation Results

Dataset Rain200H Rain200L DDN-Data DID-Data SPA-Data AGAN-Data
PSNR 32.10 40.74 33.80 34.85 47.34 31.63
SSIM 0.9343 0.9884 0.9407 0.9401 0.9929 0.9360

Pretrained Model

Citation

If you use IDT, please consider citing:

@article{xiao2022image,
    title={Image De-raining Transformer},
    author={Xiao, Jie and Fu, Xueyang and Liu, Aiping and Wu, Feng and Zha, Zheng-Jun},
    journal={IEEE Transactions on Pattern Analysis and Machine Intelligence},
    year={2022},
    publisher={IEEE}
}

Acknowledgement

This code is based on the Uformer. The evaluation code borrows from Restormer and AttentGAN.

Contact

Please contact us if there is any question([email protected]).

About

Official repository for the paper "Image Deraining Transformer".

Topics

Resources

License

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published