An all in one solution for adding Temporal Stability to a Stable Diffusion Render via an automatic1111 extension

You must install FFMPEG to path before running this

You can find a demonstration run through with a single batch here:
https://twitter.com/CiaraRowles1/status/1645923461343363072

And a batch demonstration here:
https://mobile.twitter.com/CiaraRowles1/status/1646458056803250178

Ebsynth tutorial:
https://twitter.com/CiaraRowles1/status/1648462374125576192
NOTE: EBSYNTH DOES NOT REGISTER THE KEYFRAMES IF YOU USE ABOVE 20.

Ebsynth split frames tutorial:
https://www.youtube.com/watch?v=z3YNHiuvxyg&ab_channel=CiaraRowles

Example results you can get:

Q: My video has smearing.

A: Use a higher fps and/or lower batchnumber, the closer together the keyframes the less artifacts.

Q: Stable diffusion cannot be turned on after installing this Extension. ModuleNotFoundError: No module named 'tqdm.auto'

A: Because the dependency currently used by this plug-in uses an old version of tqdm, an error occurs. The short-term solution is to manually install a new version(4.66.1) of tqdm.

pip install tqdm==4.66.1

More information

Written with reference to this web page teaching and my own(cocomine) experience

Open Extensions tab > Install from URL > Paste follow link in URL for extension’s git repository > Click Install

https://github.com/CiaraStrawberry/TemporalKit

sudo apt install ffmpeg

sudo pacman -S ffmpeg

Download and install from https://ffmpeg.org/download.html Make sure to add ffmpeg to your PATH. Learn more: https://www.wikihow.com/Install-FFmpeg-on-Windows

Create a folder in your desired location. This folder will be used to store the files that need to be processed. Prepare the video you need to use (referred to as the original video in subsequent teaching), and understand the format of the original video, such as resolution and frame.

1. Open Temporal-Kit Tab on Top.
2. Open Pre-Process Tab.
3. Drag & Drop the original video into the Input Video.
4. Set fps to the frame rate of the original video.
5. Set frames per keyframe to the number of frames between each keyframe. For example, if the original video is 30fps and you set it to 10, then 3 keyframes will be generated per second, and the rest will be estimated.
6. Set Side to the square root of the number of frames per plate. For example, if you set it to 2, 4 plates(2x2) will be generated, 3, 9 plates(3x3), 4, 16 plates(4x4).
7. Set Height Resolution to the size of each plate. It is strongly recommended that you set this to a multiple of your side variable.
   For example, if you want to generate 4 plates and set the side to 2, each plate high 512, then you need to set the height resolution to 1024(512x2).
8. Set Target Folder to the folder you created in step 4.
9. Open Batch Settings Tab.
10. Tick the Batch Run.
11. Open EBsynth Tab.
12. Tick the Split Video

When you complete the above steps you should see a structure similar to this in the folder you specified (depending on the length of your video)

You will see a structure like this in the video clips divided into folders named by numbers.

If you encounter out of memory issue in the next img2img step, reduce the side or Height Resolution parameters.

Let’s deal with the folder named 0 first.
Go to the Img2img page. Switch to the Batch tab. Set the following parameters:

Input directory: The name of your target directory with \input appended. E.g. YOUR_FOLDER_PATH_IN_SETP_4\0\input
Output directory: Similarly but with \output appended. E.g. YOUR_FOLDER_PATH_IN_SETP_4\0\output

Enter a prompt and a negative prompt like txt2img.
Sampling method: DPM++2M Karras
Sampling steps: 20
CFG scale: 7
Denoising strength: 0.5 (adjust accordingly)

The above parameters can be changed as needed.

In ControlNet (Unit 0) section, set:

• Enable: Yes
• Pixel Perfect: Yes
• ControlType: Tile
• Preprocessor: tile_resample
• Model: control_xxxx_tile

Press Generate. After it is done, you will find the image in the batch output folder.

Make sure to open the image in full size and inspect the details in full size. Make sure they look sharp and have a consistent style.

If you want to obtain high-resolution images, please put the output images back into img2img, and adjust resizd by 1.5-2.0, Denoising strength 0.3-0.4, and then generate.

Go to Temporal-Kit page and switch to the Ebsynth-Process tab.

Input Folder: Put in the same target folder path you put in the Pre-Processing page. E.g. YOUR_FOLDER_PATH_IN_SETP_4\0

Click read last_settings. If your input folder is correct, the video and the settings will be populated.

Click prepare ebsynth. After it is done, you should see the keys folder populated with your stylized keyframes, and the frames folder populated with your images.

Please note that this program does not generate .ebs files. When your images are imported into the program, they will be automatically populated.

Now open the EbSynth program.

Open the File Explorer and navigate to the folder your creation in step4. You should folder like the ones showed below. We need the keys folder and the frames folder for EbSynth.

Drag the keys folder from the File Explorer and drop it to the Keyframes field in EbSynth.
Drag the frames folder from the File Explorer and drop it to the frames field in EbSynth.

Click Run All and wait for them to complete. When it is done, you should see a series of out_##### directories generated in the target project folder.

Please download the program from the official website.
https://ebsynth.com/

Now go back to AUTOMATIC1111(webUI). You should still be on the Temporal Kit page and Ebsynth-Process tab.

Click recombine ebsynth and you are done!

Look how smooth the video is. With some tweaking, you can probably make it better!

Repeat steps 6 to 9. Process other folders (1,2,3,4,...)

Use a video splicing program to merge segmented videos

• set up diffusion based upscaling for the plates output
• get the img2img button working with batch processing.
• add a check to see if the output folder was added.
• fix that weird shutdown error it gives after running
• hook up to the api.
• flowmaps from game engine export\import support

Thanks to RAFT for the optical flow system.