This repository contains the Argo Workflow pipeline for Stable Diffusion Inference on Minikube cluster. It also contains argo workflow config for toy fan-in/fan-out pipeline.
Repository contains script to setup minikube cluster with argo workflow installed.
For stable diffusion pipeline the following repo is used: stable-diffusion-docker.
Cluster setup is defined in create_minikube_cluster.sh.
The stable diffusion workflow is defined in parallel_inference.yml.
It is executed by running argo -n argo submit argo/parallel_inference.yml
.
The fan-in/fan-out workflow is defined in fan_in_fan_out.yml.
It is executed by running argo -n argo submit argo/fan_in_fan_out.yml
.
The code was tested on MacOs Intel with minikube v1.31.1 and kubectl v1.27.0. Stable diffusion pipeline doesn't work on MacOs M1, though minikube setup and fan-in-fan-out example work.
To run fan-in/fan-out you'll need minimum of resources, but for stable diffusion you'll need at least 8GB of RAM.
- Sign up to huggingface.co and get hugging face token.
- Sign up to hub.docker.com, create private repository and get docker token.
- To install minikube you'll need driver. My personal preference is having Docker installed and running, so minikube use it as driver
- Install kubectl and minikube. It is better too install latest versions for both.
- In create_minikube_cluster.sh check minikube cluster parameters and replace if needed:
export MINIKUBE_CLUSTER_RAM="17GB"
export MINIKUBE_CLUSTER_CPUS="4"
export MINIKUBE_CLUSTER_DISK_SIZE="40GB"
- In create_minikube_cluster.sh replace placeholders with your values:
export DOCKER_USERNAME=INSERT_YOUR_VALUE_HERE
export DOCKER_TOKEN=INSERT_YOUR_VALUE_HERE
export DOCKER_PASSWORD=INSERT_YOUR_VALUE_HERE
export DOCKER_EMAIL=INSERT_YOUR_VALUE_HERE
export HUGGINGFACE_TOKEN=INSERT_YOUR_VALUE_HERE
bash create_minikube_cluster.sh
This command will create minikube cluster and argo namespace. Then it will install argo to the cluster and setup argo workflow executor. It will create secrets with docker and huggingface credentials to use in workflow. It will set admin right to default account. Then it will create PVC (Persistent Volume Claim) for huggingface models caching. Finally it will install Minio artifact repository to cluster and set it as default artifact repository for argo workflow.
- Open Argo UI in browser. Run the following command. Then go to
https://localhost:2746
in your browser. If your browser saying it is not safe, still proceed
argo -n argo server --auth-mode=server
- Open Minio UI in browser. Run the following command. Then go to
https://localhost:9000
in your browser. If your browser saying it is not safe, still proceed
kubectl -n argo get service argo-artifacts
kubectl -n argo port-forward service/argo-artifacts 9000:9000
- To login to Minio, get access key and secrets key with following commands.
export ACCESS_KEY=$(kubectl get secret argo-artifacts --namespace argo -o jsonpath="{.data.accesskey}" | base64 --decode)
export SECRET_KEY=$(kubectl get secret argo-artifacts --namespace argo -o jsonpath="{.data.secretkey}" | base64 --decode)
- In Minio UI login with access key and secret key. Create bucket with name that you set in create_minikube_cluster.sh script in DEFAULT_ARTIFACT_REPOSITORY variable.
This is argo workflow example with fan-in/fan-out pattern to autoscale workflow step.
In this workflow we run cowsay
command in parallel with different characters option saying input_phrase
parameter.
It takes 2 parameters as input:
number_of_doodles
- number of characters to run cowsay withinput_phrase
- phrase to say
To run with default parameters, run argo -n argo submit argo/fan_in_fan_out.yml
.
To change any of parameters, run argo -n argo submit argo/fan_in_fan_out.yml -p number_of_doodles="10" -p input_phrase="Hello Moon"
.
This is argo workflow example with stable diffusion inference pipeline. Pipeline build docker image with stable diffusion inference and push it to docker hub. Then it runs inference on input text prompts in parallel docker containers.
By default, it uses cpu for inference, but you can switch to gpu by changing device
parameter to cuda
in
docker-entrypoint.py
script command in parallel_inference.yml.
To generate images dreamlike-art/dreamlike-photoreal-2.0
is used by default. And each prompt will produce 2 images.
It takes 4 parameters as input:
repo
- link to github repo to build docker image from. We use stable-diffusion-docker.image
- docker image name to push to docker hub in a formatDOCKER_LOGIN/PRIVATE_REPO_NAME:TAG
.if_build_image
-true
orfalse
value. Iffalse
will build and push image as first step of workflow. Iftrue
will skip image building and pull existing image.input_prompts
- prompts to run inference on. Should be string divided by;
symbol. Each prompt will produce 2 images.
Before running check parallelism
parameter in parallel_inference.yml. It is set to 2
by default,
but it is better to set it according to your resources. For example, I was able to run it with parallelism=2
on my MacOs Intel with 15GB of RAM and 4 CPU cores,
though original repo stated that one docker image requires 8GB of RAM and 4 CPU cores, so maybe you'll need to set parallelism=1
to run it on your machine.
To run with default parameters, run argo -n argo submit argo/parallel_inference.yml -p image="DOCKER_LOGIN/PRIVATE_REPO_NAME:TAG"
.