This is the official implementation for Cylin-Painting (IEEE TIP 2024).

Kang Liao, Xiangyu Xu, Chunyu Lin, Wenqi Ren, Yunchao Wei, Yao Zhao

Problem

Given a fixed field of view (FoV) image, Cylin-Painting aims to extrapolate a 360° texturally seamless and semantically consistent panoramic image.

Features

• First effort to analyze the essential difference between image inpainting and image outpainting theoretically and experimentally
• Efficiently fuse the different spatial arrangements of the input image, which also enables a seamless 360° panoramic image extrapolation
• Make an early attempt to systematically describe the strengths and limitations of positional encoding in CNNs. Furthermore, we tame the cylinder convolution with a novel learnable positional encoding, which essentially improves the generation results
• Our method can serve as a plug-and-play module and can flexibly extend to other 360° panoramic vision tasks including low-level tasks and high-level tasks

• 20240101: Cylin-Painting is published at TIP.
• 20240118: The implementation guidance is released.
• 20240202: Release both training and inference codes.
• 20240202: Release pre-trained weights on panoramic datasets.

Using the virtual environment (conda) to run the code is recommended.

conda create -n cylin_painting python=3.6
conda activate cylin_painting
pip install -r requirements.txt

We explored the panoramic image outpainting tasks on three panoramic image datasets: SUN360 [1], Matterport3D [2], and 360SP [3], including the indoor and outdoor scenes. For each dataset, we consider the image outpainting case with the resolution of 256 × 256 → 512 × 256.

Download the pretrained model here and put it into the .\checkpoint folder.

Customize the paths of training datasets and saving checkpoints, GPU id, and run:

sh scripts/train.sh

Customize the paths of checkpoint and test set, and run:

sh scripts/test.sh

While designed for panoramic image outpainting, our method can effectively extend to other panoramic vision tasks as a plug-and-play module, including high-level and low-level tasks.

[1] Xiao, J., Ehinger, K.A., Oliva, A. and Torralba, A., 2012, June. Recognizing scene viewpoint using panoramic place representation. In 2012 IEEE Conference on Computer Vision and Pattern Recognition.
[2] Chang, A., Dai, A., Funkhouser, T., Halber, M., Niessner, M., Savva, M., Song, S., Zeng, A. and Zhang, Y., 2017. Matterport3d: Learning from rgb-d data in indoor environments. arXiv preprint arXiv:1709.06158.
[3] Chang, S.H., Chiu, C.Y., Chang, C.S., Chen, K.W., Yao, C.Y., Lee, R.R. and Chu, H.K., 2018. Generating 360 outdoor panorama dataset with reliable sun position estimation. In SIGGRAPH Asia 2018 Posters.

If our solution is useful for your research, please consider citing:

@ARTICLE{cylin_painting,
  author={Liao, Kang and Xu, Xiangyu and Lin, Chunyu and Ren, Wenqi and Wei, Yunchao and Zhao, Yao},
  journal={IEEE Transactions on Image Processing}, 
  title={Cylin-Painting: Seamless 360° Panoramic Image Outpainting and Beyond}, 
  year={2024},
  volume={33},
  pages={382-394}
}