Not working with k0s

[ianb-mp]

What steps did you take and what happened:

After install, I am unable to create PVC using zfs-localpv storageclass. I notice that CSI driver is not registering:

 $ kubectl get csinode
NAME           DRIVERS   AGE
bne-lab-vr-3   0         17h

I install via helm chart like this:

helm install openebs --namespace openebs openebs/openebs --create-namespace \
  --set engines.replicated.mayastor.enabled=false \
  --set zfs-localpv.zfsNode.kubeletDir=/var/lib/k0s/kubelet

And use this storageclass:

storageclass.yaml

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
  name: openebs-zfspv
parameters:
  recordsize: "128k"
  compression: "off"
  dedup: "off"
  fstype: "zfs"
  poolname: "zfspv-pool"
provisioner: zfs.csi.openebs.io

What did you expect to happen:

zfs-localpv should work with k0s

The output of the following commands will help us better understand what's going on:
(Pasting long output into a GitHub gist or other Pastebin is fine.)

• kubectl logs -f openebs-zfs-controller-[xxxx] -n openebs -c openebs-zfs-plugin - see: https://gist.github.com/ianb-mp/98660a8057a8c8c53cd8f7e84740a3ac#file-openebs-zfs-controller_openebs-zfs-plugin-log
• kubectl logs -f openebs-zfs-node-[xxxx] -n openebs -c openebs-zfs-plugin - see: https://gist.github.com/ianb-mp/98660a8057a8c8c53cd8f7e84740a3ac#file-openebs-zfs-localpv-node_openebs-zfs-plugin-log
• kubectl get pods -n openebs

NAME                                              READY   STATUS    RESTARTS   AGE
openebs-zfs-localpv-node-5k7wl                    2/2     Running   0          6m7s
openebs-localpv-provisioner-6b8bff68bd-tdvxh      1/1     Running   0          6m7s
openebs-zfs-localpv-controller-6f879b95b-r6k9d    5/5     Running   0          6m7s
openebs-lvm-localpv-node-jl7qt                    2/2     Running   0          6m7s
openebs-lvm-localpv-controller-59cd69bd56-lmwrk   5/5     Running   0          6m7s

• kubectl get zv -A -o yaml (none created)

Anything else you would like to add:

• I see the same issue with lvm & hostpath drivers.
• I have installed Longhorn storage driver on the same host (just to compare) and that worked fine.

I understand this could be k0s issue, but I thought probably better to create issue here first before going to k0s.

Environment:

• LocalPV-ZFS version: v2.5.1
• Kubernetes version (use kubectl version):

$ kubectl version
Client Version: v1.30.0
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.29.4+k0s

• Kubernetes installer & version: k0sctl v0.17.5
• Cloud provider or hardware configuration: bare metal, x86_64
• OS (e.g. from /etc/os-release): Rocky Linux 9.3

[w3aman]

Can you try with appending / at the end in kubelet path like this /var/lib/k0s/kubelet/ ?

[ianb-mp]

It's working now with trailing /, thanks! The README should be updated to reflect this, and ideally the code should be modified so that it can handle path with trailing slash or without.

[secustor]

I have found this after some hours of googling and it fixed my issues too.
These have been my symptoms, so it is more easily findable by other users:

Setup:

• k0s: v1.30.0+k0s.0
• OpenEBS ( Helm chart ): 4.0.1
  values before fix:

       zfs-localpv:
         enabled: true
         zfsNode:
           # needs to be overwritten as else no data regarding free storage is available in K8
           kubeletDir: /var/lib/k0s/kubelet

On the pending pod:

Warning  FailedScheduling  25m (x5 over 42m)  default-scheduler  0/1 nodes are available: 1 node(s) did not have enough free storage. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.