ComfyUI reference controlnet support

[jax-explorer]

Why is reference controlnet not supported in ControlNet?

[Kosinkadink]

What do you mean?

[jax-explorer]

https://github.com/Mikubill/sd-webui-controlnet

In Controlnet there is a reference Controlnet, which references a picture, but I don't find it in ComfyUI.

[Kosinkadink]

I added ReferenceCN support a couple weeks ago. The input images must be put through the ReferenceCN Preprocessor, with the latents being the same size (h and w) that will be going into the KSampler.

I also improved on the auto1111 implementation by adding a true strength control. The ref_weight is the same as auto1111's controlnet strength, the style_fidelity is the same as auto1111's style_fidelity, and the strength of the controlnet (whether that be on the Apply ControlNet node or on the strength inputs on the finetune node) is granular control over the strength. The finetune node also allows you to choose separate values for attn and adain.

[jax-explorer]

@Kosinkadink It's really awesome, a lot of times ReferenceCN is very useful and a lot of people think that IPadapter can replace it, but they are totally different. I've used the node before but it reported an error, I think there was a problem with some of the width and height settings, I'll try it again soon, thanks for the code.

[jax-explorer]

I tried this ReferenceCN and it doesn't seem to do anything, here is my test workflow

[Kosinkadink]

I can't replicate your issue. Can you confirm that if you bypass the Apply Advanced ControlNet that the output image does not change (which would mean no ref stuff happens)? Just tested both the KSampler and KSampler (Advanced) and they both work fine for me.

(this second image here contains the workflow that you can drag into ComfyUI to load)

[jax-explorer]

It serves some purpose, but does not achieve character consistency. It still doesn't look like there's a good program for role consistency.

[Kosinkadink]

The comfy implementation works basically identically to auto1111. Can you send me links to the exact models you're using, the exact workflow, and the exact input images? I will try to run your stuff on my end.

ReferenceCN will never be exact as it's just the nature of how it works, but I want to confirm its working as intended on your end.

[jax-explorer]

You are treating the open source project very positively and this is commendable.

ok, I am using the model https://civitai.com/models/7240/meinamix and the input images are: https://image.civitai.com/xG1nkqKTMzGDvpLrqFT7WA/905ade32-70b9-42e0-9d83-4b3393aee020/original=true/81EB23FAB3E910AC5695CBA24FB09EB43E3D7EBA968E908AA0F3F6A81550F736.jpeg "positive_prompt": "high resolution, masterpiece", "negative_prompt":"(worst quality, low quality:1.4), monochrome, zombie, (interlocked fingers:1.2),
"negative_prompt": "(worst quality, low quality:1.4), monochrome, zombie, (interlocked fingers:1.2)", I'll post the full workflow later, but the information described above should suffice.

Thank you for your positive response.

[MoonMoon82]

Does anyone else got an issue with it now?

!!! Exception during processing!!! 'NoneType' object has no attribute '__get__'
Traceback (most recent call last):
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 63, in refcn_sample
    injection_holder = InjectionBasicTransformerBlockHolder(block=module, idx=i)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 433, in __init__
    self.original_forward = block._forward
                            ^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1709, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'BasicTransformerBlock' object has no attribute '_forward'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1344, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1314, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample
    return original_sample(*args, **kwargs)  # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 313, in motion_sample
    return orig_comfy_sample(model, noise, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 134, in refcn_sample
    model.model.diffusion_model.forward = reference_injections.diffusion_model_orig_forward.__get__(model.model.diffusion_model, type(model.model.diffusion_model))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '__get__'

Prompt executed in 0.17 seconds

[SSA4]

I get the samer error too. The Reference Only seems to be broken.

[Kosinkadink]

I'll take a look in a few hours. On the surface nothing changed that should have broken it, but could be an odd side effect of the recent uncond_multiplier hacky implementation.

[amiibot]

何改变应该会破坏它，但这可能是

so do I.It can not work

Does anyone else got an issue with it now?

!!! Exception during processing!!! 'NoneType' object has no attribute '__get__'
Traceback (most recent call last):
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 63, in refcn_sample
    injection_holder = InjectionBasicTransformerBlockHolder(block=module, idx=i)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 433, in __init__
    self.original_forward = block._forward
                            ^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\python_embeded\Lib\site-packages\torch\nn\modules\module.py", line 1709, in __getattr__
    raise AttributeError(f"'{type(self).__name__}' object has no attribute '{name}'")
AttributeError: 'BasicTransformerBlock' object has no attribute '_forward'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 151, in recursive_execute
    output_data, output_ui = get_output_data(obj, input_data_all)
                             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 81, in get_output_data
    return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\execution.py", line 74, in map_node_over_list
    results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1344, in sample
    return common_ksampler(model, seed, steps, cfg, sampler_name, scheduler, positive, negative, latent_image, denoise=denoise)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\nodes.py", line 1314, in common_ksampler
    samples = comfy.sample.sample(model, noise, steps, cfg, sampler_name, scheduler, positive, negative, latent_image,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Impact-Pack\modules\impact\sample_error_enhancer.py", line 9, in informative_sample
    return original_sample(*args, **kwargs)  # This code helps interpret error messages that occur within exceptions but does not have any impact on other operations.
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-AnimateDiff-Evolved\animatediff\sampling.py", line 313, in motion_sample
    return orig_comfy_sample(model, noise, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "E:\StableDiffusion\ComfyUI_windows_portable\ComfyUI\custom_nodes\ComfyUI-Advanced-ControlNet\adv_control\control_reference.py", line 134, in refcn_sample
    model.model.diffusion_model.forward = reference_injections.diffusion_model_orig_forward.__get__(model.model.diffusion_model, type(model.model.diffusion_model))
                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: 'NoneType' object has no attribute '__get__'

Prompt executed in 0.17 seconds

It seems to be caused by a comfyui version update, I have two copies of comfyui and it doesn't report an error on the cd07340d version, while the latest version gives an error.

[Kosinkadink]

I've found the source of the bug - it is indeed due to the latest commit to ComfyUI. When I get home in a few hours, I'll make an update that will make RefCN work with both older and current ComfyUI

[ikcikoR]

Can confirm the bug is happening and reverting by just a single commit currently fixes it.

[Kosinkadink]

RefCN attn is now compatible with newest ComfyUI, while maintaining backwards compatibility with previous ComfyUI versions.