When using SDXL+animatediff+Tgate, the following error occurs:

[hahy36]

Error occurred when executing SamplerCustom:

Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated : 12.81 GiB
Requested : 480.00 MiB
Device limit : 14.75 GiB
Free (according to CUDA): 27.06 MiB
PyTorch limit (set by user-supplied memory fraction)
: 17179869184.00 GiB

File "/kaggle/working/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/kaggle/working/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/kaggle/working/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/kaggle/working/ComfyUI/comfy_extras/nodes_custom_sampler.py", line 374, in sample
samples = comfy.sample.sample_custom(model, noise, cfg, sampler, sigmas, positive, negative, latent_image, noise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise_seed)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 419, in motion_sample
latents = orig_comfy_sample(model, noise, *args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/sample.py", line 42, in sample_custom
samples = comfy.samplers.sample(model, noise, positive, negative, cfg, model.load_device, sampler, sigmas, model_options=model.model_options, latent_image=latent_image, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 663, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 650, in sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 629, in inner_sample
samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 534, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-sampler-lcm-alternative/sampler_lcm_alt.py", line 32, in sample_lcm_cycle
return sample_lcm_backbone(model, x, sigmas, extra_args, callback, disable, noise_sampler, loop_control, ancestral)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-sampler-lcm-alternative/sampler_lcm_alt.py", line 38, in sample_lcm_backbone
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 272, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 616, in call
return self.predict_noise(*args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 619, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 456, in evolved_sampling_function
cond_pred, uncond_pred = sliding_calc_conds_batch(model, [cond, uncond], x, timestep, model_options)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 593, in sliding_calc_conds_batch
sub_conds_out = calc_cond_uncond_batch_wrapper(model, sub_conds, sub_x, sub_timestep, model_options)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 653, in calc_cond_uncond_batch_wrapper
return comfy.samplers.calc_cond_batch(model, conds, x_in, timestep, model_options)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
File "/kaggle/working/ComfyUI/comfy/model_base.py", line 97, in apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 885, in forward
h = forward_timestep_embed(module, h, emb, context, transformer_options, output_shape, time_context=time_context, num_video_frames=num_video_frames, image_only_indicator=image_only_indicator)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 130, in forward_timestep_embed
x = layer(x, emb)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 230, in forward
return checkpoint(
File "/kaggle/working/ComfyUI/comfy/ldm/modules/diffusionmodules/util.py", line 191, in checkpoint
return func(*inputs)
File "/kaggle/working/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 243, in _forward
h = self.in_layers(x)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/ops.py", line 94, in forward
return super().forward(*args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 171, in groupnorm_mm_forward
input = group_norm(input, self.num_groups, weight, bias, self.eps)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/functional.py", line 2561, in group_norm
return torch.group_norm(input, num_groups, weight, bias, eps, torch.backends.cudnn.enabled)

[hahy36]

Error occurred when executing SamplerCustom:

Allocation on device 0 would exceed allowed memory. (out of memory)
Currently allocated : 13.74 GiB
Requested : 640.00 MiB
Device limit : 14.75 GiB
Free (according to CUDA): 27.06 MiB
PyTorch limit (set by user-supplied memory fraction)
: 17179869184.00 GiB

File "/kaggle/working/ComfyUI/execution.py", line 151, in recursive_execute
output_data, output_ui = get_output_data(obj, input_data_all)
File "/kaggle/working/ComfyUI/execution.py", line 81, in get_output_data
return_values = map_node_over_list(obj, input_data_all, obj.FUNCTION, allow_interrupt=True)
File "/kaggle/working/ComfyUI/execution.py", line 74, in map_node_over_list
results.append(getattr(obj, func)(**slice_dict(input_data_all, i)))
File "/kaggle/working/ComfyUI/comfy_extras/nodes_custom_sampler.py", line 374, in sample
samples = comfy.sample.sample_custom(model, noise, cfg, sampler, sigmas, positive, negative, latent_image, noise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=noise_seed)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-Advanced-ControlNet/adv_control/control_reference.py", line 47, in refcn_sample
return orig_comfy_sample(model, *args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 419, in motion_sample
latents = orig_comfy_sample(model, noise, *args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/sample.py", line 42, in sample_custom
samples = comfy.samplers.sample(model, noise, positive, negative, cfg, model.load_device, sampler, sigmas, model_options=model.model_options, latent_image=latent_image, denoise_mask=noise_mask, callback=callback, disable_pbar=disable_pbar, seed=seed)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 663, in sample
return cfg_guider.sample(noise, latent_image, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 650, in sample
output = self.inner_sample(noise, latent_image, device, sampler, sigmas, denoise_mask, callback, disable_pbar, seed)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 629, in inner_sample
samples = sampler.sample(self, sigmas, extra_args, callback, noise, latent_image, denoise_mask, disable_pbar)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 534, in sample
samples = self.sampler_function(model_k, noise, sigmas, extra_args=extra_args, callback=k_callback, disable=disable_pbar, **self.extra_options)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-sampler-lcm-alternative/sampler_lcm_alt.py", line 32, in sample_lcm_cycle
return sample_lcm_backbone(model, x, sigmas, extra_args, callback, disable, noise_sampler, loop_control, ancestral)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/utils/contextlib.py", line 115, in decorate_context
return func(*args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-sampler-lcm-alternative/sampler_lcm_alt.py", line 38, in sample_lcm_backbone
denoised = model(x, sigmas[i] * s_in, **extra_args)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 272, in call
out = self.inner_model(x, sigma, model_options=model_options, seed=seed)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 616, in call
return self.predict_noise(*args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 619, in predict_noise
return sampling_function(self.inner_model, x, timestep, self.conds.get("negative", None), self.conds.get("positive", None), self.cfg, model_options=model_options, seed=seed)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 456, in evolved_sampling_function
cond_pred, uncond_pred = sliding_calc_conds_batch(model, [cond, uncond], x, timestep, model_options)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 593, in sliding_calc_conds_batch
sub_conds_out = calc_cond_uncond_batch_wrapper(model, sub_conds, sub_x, sub_timestep, model_options)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 653, in calc_cond_uncond_batch_wrapper
return comfy.samplers.calc_cond_batch(model, conds, x_in, timestep, model_options)
File "/kaggle/working/ComfyUI/comfy/samplers.py", line 218, in calc_cond_batch
output = model.apply_model(input_x, timestep, **c).chunk(batch_chunks)
File "/kaggle/working/ComfyUI/comfy/model_base.py", line 97, in apply_model
model_output = self.diffusion_model(xc, t, context=context, control=control, transformer_options=transformer_options, **extra_conds).float()
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/ldm/modules/diffusionmodules/openaimodel.py", line 850, in forward
h = forward_timestep_embed(module, h, emb, context, transformer_options, time_context=time_context, num_video_frames=num_video_frames, image_only_indicator=image_only_indicator)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/sampling.py", line 132, in forward_timestep_embed
x = layer(x, context)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/motion_module_ad.py", line 673, in forward
return self.temporal_transformer(input_tensor, encoder_hidden_states, attention_mask, self.view_options, mm_kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/motion_module_ad.py", line 891, in forward
hidden_states = block(
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/custom_nodes/ComfyUI-AnimateDiff-Evolved/animatediff/motion_module_ad.py", line 1084, in forward
hidden_states = self.ff(self.ff_norm(hidden_states)) + hidden_states
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/ldm/modules/attention.py", line 83, in forward
return self.net(x)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/container.py", line 217, in forward
input = module(input)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1511, in _wrapped_call_impl
return self._call_impl(*args, **kwargs)
File "/kaggle/venv/venv/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1520, in _call_impl
return forward_call(*args, **kwargs)
File "/kaggle/working/ComfyUI/comfy/ldm/modules/attention.py", line 63, in forward
return x * F.gelu(gate)

[JettHu]

Is it normal when t-gate is not used? Because I see that the error location is not on t-gate.

[hahy36]

I also find it strange, but after testing two large models, I discovered that as long as Tgate is disabled, the error no longer occurs.

[hahy36]

It might also be due to a plugin called Style Aligned, there are some compatibility issues.But I can't give up on Style Aligned.

[JettHu]

Can you provide the Style Aligned plugin link.

[hahy36]

https://github.com/brianfitzgerald/style_aligned_comfy

[hahy36]

I might not have expressed myself clearly, let me add some clarification:
SD1.5+animatediff+Tgate=√
SDXL+animatediff+Tgate=×

[hahy36]

Or.
SD1.5+animatediff+styel_aligned+Tgate=√
SDXL+animatediff+styel_aligned+Tgate=×

[JettHu]

Thanks for the information, I will take a look at this issue

[JettHu]

I might not have expressed myself clearly, let me add some clarification: SD1.5+animatediff+Tgate=√ SDXL+animatediff+Tgate=×

It seems to be a problem with animatediff. Both animatediff and T-GATE hook comfy.samplers.sampling_function. So in fact, the part with the greatest acceleration of T-Gate (skipping the cfg part in the sampling) actually does not work. It become equivalent to only cache attntion result. However, the OOM of the GPU memory should be caused by too much attention being cached.

• The current implementation is not fully compatible with animatediff.
• GPU OOM is relatively easy to solve. I can quickly update the version, but the performance is not as good as txt2img/img2img.

Thank you for your issue. Compatibility with animatediff will be on the agenda.

[JettHu]

@hahy36 hey, I updated repo yestoday. You can try the new node and new parameters.

[hahy36]

It has been resolved. Great job!