中文

An unofficial ComfyUI custom node for ZeST(Zero-Shot Material Transfer from a Single Image)

Given an input image (e.g., a photo of an apple) and a single material exemplar image (e.g., a golden bowl), ZeST can transfer the gold material from the exemplar onto the apple with accurate lighting cues while making everything else consistent.

Refer to zest_workflow.json in this repo for quick start.

This node implenment the red rect area of paper figure. the illumination transfer, and make it an ComfyUI node. You can use any IPA and controlnet node as your wish.

The original ZeST demo code use IPA to extract the style of material, and control for depth information. Then, change the original image with illumination informaion of subject for better lighting style.

in ComfyUI Manager or git clone to ComfyUI/custom_nodes

After restart ComfyUI, the following custom node will be available.

• ControlNet
• IP-Adaptor
• Segment Anything Model

• INPUT:

  • target_image : the original image for inpaint
  • subject_mask : the mask for inpaint, this mask will be also used as input of inpaint node
  • brighter : default is 1, means no change
    • value < 1 , means darker the subject, useful when subject in high light
    • value > 1, means brighter the subject, useful when subject in dark light

• OUTPUT:

  • IMAGE : image with subject in luminosity(grey) mode. used as input for inpaint node

1. for output, the subject in middle grey is prefered. for high light material, dark it; otherwise, brighter it.
2. for material image, suggest remove all backgroup, just keep the material you want.
3. use IP-Adaptor strength, controlnet strength, brighter to control the output 1st.

You can download the zest_workflow.json from this repo for quickly start.

• ZeST: Zero-Shot Material Transfer from a Single Image Paper
• ZeST official DEMO
• ZeST site
• ZeST official video

@article{cheng2024zest,
  title={ZeST: Zero-Shot Material Transfer from a Single Image},
  author={Cheng, Ta-Ying and Sharma, Prafull and Markham, Andrew and Trigoni, Niki and Jampani, Varun},
  journal={arXiv preprint arXiv:2404.06425},
  year={2024}
}