Dcam is an experimental project that applies the Stable Diffusion image-to-image process to a real-time video input.

Dcam uses Apple's Core ML port of Stable Diffusion (ml-stable-diffusion) to run the pipeline on macOS. It also uses the NDI protocol to stream video and controller inputs from iPhone to Mac over LAN.

I used this project in Channel 23 and other events for concert visuals.

• Unity 2023.2
• macOS 14 Sonoma
• Mac computer with many GPU cores (M2 Max or later models are recommended)
• iPhone with triple lenses (iPhone 13 Pro or later models)
• Network connection (or USB connection; see below)

You can connect an iPhone and a Mac with a USB cable to establish a VLAN between them, which is handy for keeping a robust connection in a concert venue.

This project uses a Core ML Stable Diffusion model with a landscape aspect ratio (640x384). You can download the model from the Hugging Face repository below.

https://huggingface.co/keijiro-tk/coreml-stable-diffusion-2-1-base-640x384

The Stable Diffusion image-to-image process has a few seconds of latency, even on a powerful Mac computer. I tried hiding this latency by inserting flipbook-like effects.

You can reduce the latency by using a PC with a high-performance GPU, but bringing a bulky and heavy PC into a venue is troublesome. I prefer PCs for research purposes but MacBooks for on-site work.

iPhone is handy for video input and remote control. I can connect them using a USB extender cable (USB repeater) and establish a robust NDI connection.