Skip to content
/ CLIPNeRF Public

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

License

Apache-2.0 license
259 stars 15 forks Branches Tags Activity
Star
Notifications You must be signed in to change notification settings

cassiePython/CLIPNeRF

BranchesTags

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

7 Commits
imgs
imgs
 
 
supplyment
supplyment
 
 
LICENSE
LICENSE
 
 
README.md
README.md
 
 

Repository files navigation

CVPR2022: CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

Project Page | Paper (ArXiv)

Can Wang1, Menglei Chai2, Mingming He3, Dongdong Chen4, Jing Liao1
1City University of Hong Kong, 2Creative Vision, Snap Inc., 3USC Institute for Creative Technologies, 4Microsoft Cloud AI

Abstract

We present CLIP-NeRF, a multi-modal 3D object manipulation method for neural radiance fields (NeRF). By leveraging the joint language-image embedding space of the recent Contrastive Language-Image Pre-Training (CLIP) model, we propose a unified framework that allows manipulating NeRF in a user-friendly way, using either a short text prompt or an exemplar image. Specifically, to combine the novel view synthesis capability of NeRF and the controllable manipulation ability of latent representations from generative models, we introduce a disentangled conditional NeRF architecture that allows individual control over both shape and appearance. This is achieved by performing the shape conditioning via applying a learned deformation field to the positional encoding and deferring color conditioning to the volumetric rendering stage. To bridge this disentangled latent representation to the CLIP embedding, we design two code mappers that take a CLIP embedding as input and update the latent codes to reflect the targeted editing. The mappers are trained with a CLIP-based matching loss to ensure the manipulation accuracy. Furthermore, we propose an inverse optimization method that accurately projects an input image to the latent codes for manipulation to enable editing on real images. We evaluate our approach by extensive experiments on a variety of text prompts and exemplar images and also provide an intuitive interface for interactive editing.

Supplyment

I add the code of the single NeRF color editing in the supplyment directory.

About

CLIP-NeRF: Text-and-Image Driven Manipulation of Neural Radiance Fields

Resources

Readme

License

Apache-2.0 license
Activity

Stars

259 stars

Watchers

16 watching

Forks

15 forks
Report repository

Releases

No releases published

Packages

No packages published

Languages