ZenML on Juju with Microk8s
DISCLAIMER: This project was inspired by the Charmed MLFlow Project and also implements solutions found there.
We are assuming that you are running this tutorial on a local machine or a EC2 instance with the following specs:
-
Runs Ubuntu 22.04 or later
-
Has at least 50GB free disk space
Install MicroK8s from a snap package
sudo snap install microk8s --classic --channel=1.28/stable
Add the current user to the microk8s
group and generate configuration directory for kubectl
sudo usermod -a -G microk8s $USER
newgrp microk8s
sudo chown -f -R $USER ~/.kube
cd $HOME
mkdir .kube
cd .kube
microk8s config > config
Enable the following MicroK8s addons to configure your Kubernetes cluster with extra services needed to run Charmed Kubeflow.
microk8s enable dns hostpath-storage ingress metallb:10.64.140.43-10.64.140.49
Wait until the command
microk8s status --wait-ready
Returns
microk8s is running
NOTE: To use the MicroK8s built-in registry in the ZenML stack, please refer to the guide
To install Juju from snap, run this command:
sudo snap install juju --classic --channel=3.1/stable
On some machines there might be a missing folder which is required for juju to run correctly. Because of this please make sure to create this folder with:
mkdir -p ~/.local/share
Deploy a Juju controller to the Kubernetes we set up with MicroK8s:
microk8s config | juju add-k8s my-k8s --client
juju bootstrap my-k8s uk8sx
Create a model
juju add-model <model name>
To deploy the ZenML bundle run:
juju deploy zenml --trust
Run juju status --watch 2s
to observe the charm deployment and after all the apps reach the Active
status, the ZenML Server dashboard should be accessible as a NodePort
under
https://localhost:31375/
With the default settings for username = default
and no password.
To connect the zenml SDK
to it run
zenml connect --uri https://localhost:31375/ --username default --password ''
Install dependencies
sudo snap install charmcraft --classic
Create ZenML Charm
charmcraft pack
This step will generate a charm file zenml-server_ubuntu-20.04-amd64.charm
Deploy the ZenML server charm
juju deploy ./zenml-server_ubuntu-20.04-amd64.charm zenml-server \
--resource oci-image=$(yq '.resources."oci-image"."upstream-source"' metadata.yaml)
Deploy mysql-k8s to be used as ZenML Server backend and create relation
juju deploy mysql-k8s zenml-mysql --channel 8.0/stable
juju relate zenml-server zenml-mysql
Run juju status --watch 2s
to observe the charm deployment and after all the apps reach the Active
status, the ZenML Server dashboard should be accessible as a NodePort
under
https://localhost:31375/
With the same credentials
To use Charmed Kubeflow as a orchestrator for a ZenML stack some configurations have to be done:
To install Charmed Kubeflow follow this guide
If running Charmed Kubeflow on a EC2 instance, configure the istio-ingress-gateway
service type to NodePort
:
kubectl -n kubeflow patch svc istio-ingressgateway-workload \
-p '{"spec":{"type":"NodePort"}}'
And reconfigure dex
and oidc-gatekeeper
to point on the instance public IP:
export NODE_IP=<the instance public IP>
export NODE_PORT=$(kubectl -n kubeflow get svc istio-ingressgateway-workload -o=json | \
jq '(.spec.ports) | .[] | select(.name=="http2") | (.nodePort)')
export PUBLIC_URL="https://${NODE_IP}:${NODE_PORT}"
juju config dex-auth public-url=${PUBLIC_URL}
juju config oidc-gatekeeper public-url=${PUBLIC_URL}
Set authentication methods:
juju config dex-auth static-username="<your username>"
juju config dex-auth static-password="<your password>"
And make sure the security group allows ingress to the instance on CIDR of echo "${NODE_IP}/32"
on the echo $NODE_PORT
port
After this the Kubeflow Dashboard will be accessible under the value of echo $PUBLIC_URL
The pipeline API, required for configuring the Kubeflow Orchestrator for ZenML will have the value of:
echo "${PUBLIC_URL}/pipeline/"
Configuring Kubeflow Pipelines as a orchestrator for your ZenML workload also requires setting up an Artifact Store.
For this tutorial we will use already deployed minio
service and expose it as NodePort
kubectl -n kubeflow patch svc minio \
-p '{"spec":{"type":"NodePort"}}'
After running kubectl get svc -n kubeflow
, we should find the minio
service configuration:
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
.
.
.
minio NodePort 10.152.183.227 <none> 9000:<API port>/TCP,9001:<Dashboard port>/TCP
Now the MinIO dashboard should be accessible under:
https://localhost:<Dashboard port>
Or, when running on a EC2 with ingress configured to allow both <Dashboard port>
and <API port>
from your IP and the public IP of the machine:
echo "${PUBLIC_URL}:<Dashboard port>"
The MINIO_ACCESS_KEY
and MINIO_SECRET_KEY
in base64 can be received form Kubernetes Secrets:
kubectl get secrets minio-secret -n kubeflow -o yaml
Before using them, remember to decode the.
ZenML will require a specified bucket so use the dashboard to create one and the artifact store can be configured with:
# Store the AWS access key in a ZenML secret
zenml secret create s3_secret \
--aws_access_key_id='<DECODED_MINIO_ACCESS_KEY>' \
--aws_secret_access_key='<DECODED_MINIO_SECRET_KEY>'
# Register the MinIO artifact-store and reference the ZenML secret
zenml artifact-store register minio_store -f s3 \
--path='s3:https://<minio_bucket>' \
--authentication_secret=s3_secret \
--client_kwargs='{"endpoint_url": https://localhost:<API port>}' # or for EC2 <publicIP>:<API port>
When running a ZenML pipeline with Kubeflow Orchestrator, the client will either use the current machine docker client or preconfigured image builder to build the pipeline image and publish it to a registry. By default it is the local registry that comes with installing ZenML, but to allow an remote Kubeflow to use it, a remote registry has to be available.
For this the mentioned MicroK8s built-in registry comes in handy and can be configured with:
zenml container-registry register <NAME> \
--flavor=default \
--uri=localhost:32000 # or for EC2 <publicIP>:32000
Currently the charmed zenml-server-operator
supports ingress integration with Charmed Istio
NOTE: According to ZenML Docs:
This method has one current limitation: the ZenML UI does not support URL rewriting and will not work properly if you use a dedicated Ingress URL path. You can still connect your client to the ZenML server and use it to run pipelines as usual, but you will not be able to use the ZenML UI.
Additionally, the ZenML client does not support DEX auth - this might require configuring the server to use DEX as an external auth provider - more info here
Deploy Istio Gateway and Istio Pilot charms and configure the relation
juju deploy istio-gateway istio-ingressgateway --channel 1.17/stable --config kind=ingress --trust
juju deploy istio-pilot --channel 1.17/stable --config default-gateway=test-gateway -trust
juju relate istio-pilot istio-ingressgateway
To integrate zenml-server
with currently deployed istio-operator
run the command:
juju relate zenml-server istio-pilot
After this done the ZenML Client can connect to the server over the ingress IP found by running:
microk8s kubectl -n <your namespace> get svc istio-ingressgateway-workload -o jsonpath='{.status.loadBalancer.ingress[0].ip}'
Over the URL http:/<ingress ip>/zenml/
Check out the examples directory