Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized

[Roy214]

What happened?

Right after installing kubernetes control plane using kubeadm and weave net as network plugin the API server goes down with the below error msg.

[root@kubemaster ~]# kubectl get pods -A --watch
error: Get "https://10.74.250.78:6443/api/v1/pods?limit=500": dial tcp 10.74.250.78:6443: connect: connection refused - error from a previous attempt: read tcp 10.74.250.78:44740->10.74.250.78:6443: read: connection reset by peer
[root@kubemaster ~]#  

What did you expect to happen?

The cluster should be working fine with weave net

How can we reproduce it (as minimally and precisely as possible)?

kubeadm init --pod-network-cidr=10.10.0.0/16 --apiserver-advertise-address=10.74.250.78 --v=5

Your Kubernetes control-plane has initialized successfully!

To start using your cluster, you need to run the following as a regular user:

  mkdir -p $HOME/.kube
  sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
  sudo chown $(id -u):$(id -g) $HOME/.kube/config

Alternatively, if you are the root user, you can run:

  export KUBECONFIG=/etc/kubernetes/admin.conf

You should now deploy a pod network to the cluster.
Run "kubectl apply -f [podnetwork].yaml" with one of the options listed at:
  https://kubernetes.io/docs/concepts/cluster-administration/addons/

Then you can join any number of worker nodes by running the following on each as root:

kubeadm join 10.74.250.78:6443 --token 7l1k0v.gnzdt2i4a0hg6uaq \
	--discovery-token-ca-cert-hash sha256:9f307309f8fd825a99a30e2643a59159499d4fd78843ac14d0c78f85849da411 
[root@kubemaster ~]# 
[root@kubemaster ~]# export KUBECONFIG=/etc/kubernetes/admin.conf
[root@kubemaster ~]# 

[root@kubemaster ~]# kubectl apply -f weave-daemonset-k8s.yaml
serviceaccount/weave-net created
clusterrole.rbac.authorization.k8s.io/weave-net created
clusterrolebinding.rbac.authorization.k8s.io/weave-net created
role.rbac.authorization.k8s.io/weave-net created
rolebinding.rbac.authorization.k8s.io/weave-net created
daemonset.apps/weave-net created
[root@kubemaster ~]#

[root@kubemaster ~]# kubectl get pods -A  --watch
NAMESPACE     NAME                                 READY   STATUS             RESTARTS        AGE
kube-system   coredns-7db6d8ff4d-6t5cb             0/1     Pending            0               3m25s
kube-system   coredns-7db6d8ff4d-z9lvq             0/1     Pending            0               3m25s
kube-system   etcd-kubemaster                      1/1     Running            173 (90s ago)   58s
kube-system   kube-apiserver-kubemaster            1/1     Running            174 (72s ago)   3m59s
kube-system   kube-controller-manager-kubemaster   1/1     Running            12 (112s ago)   4m33s
kube-system   kube-proxy-jgnb6                     0/1     CrashLoopBackOff   2 (7s ago)      3m25s
kube-system   kube-scheduler-kubemaster            0/1     CrashLoopBackOff   181 (30s ago)   58s
kube-system   weave-net-fdvrk                      0/2     Pending            0               8s
kube-system   kube-scheduler-kubemaster            0/1     Running            182 (53s ago)   81s
kube-system   kube-proxy-jgnb6                     1/1     Running            3 (32s ago)     3m50s
kube-system   kube-scheduler-kubemaster            0/1     Completed          182 (61s ago)   89s
kube-system   kube-scheduler-kubemaster            0/1     CrashLoopBackOff   182 (2s ago)    90s
^C[root@kubemaster ~]# 

Error:

Jun  1 09:35:45 kubemaster kubelet[15562]: E0601 09:35:45.262058   15562 kubelet.go:2900] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Jun  1 09:35:46 kubemaster kubelet[15562]: E0601 09:35:46.177829   15562 controller.go:145] "Failed to ensure lease exists, will retry" err="Get \"https://10.74.250.78:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/kubemaster?timeout=10s\": dial tcp 10.74.250.78:6443: connect: connection refused" interval="7s"

Anything else we need to know?

No response

Kubernetes version

[root@kubemaster ~]# kubectl version
Client Version: v1.30.1

Cloud provider

VM

OS version

# On Linux:
$ cat /etc/os-release
# paste output here
$ uname -a
NAME="Red Hat Enterprise Linux"
VERSION="9.4 (Plow)"
ID="rhel"
ID_LIKE="fedora"
VERSION_ID="9.4"
PLATFORM_ID="platform:el9"
PRETTY_NAME="Red Hat Enterprise Linux 9.4 (Plow)"
ANSI_COLOR="0;31"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:redhat:enterprise_linux:9::baseos"
HOME_URL="https://www.redhat.com/"
DOCUMENTATION_URL="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9"
BUG_REPORT_URL="https://bugzilla.redhat.com/"

REDHAT_BUGZILLA_PRODUCT="Red Hat Enterprise Linux 9"
REDHAT_BUGZILLA_PRODUCT_VERSION=9.4
REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="9.4"
[root@kubemaster ~]# 
[root@kubemaster ~]# 
[root@kubemaster ~]# uname -r
5.14.0-427.18.1.el9_4.x86_64
[root@kubemaster ~]# 

Install tools

kubeadm

Container runtime (CRI) and version (if applicable)

containerd

Related plugins (CNI, CSI, ...) and versions (if applicable)

Weave Net

[k8s-ci-robot]

There are no sig labels on this issue. Please add an appropriate label by using one of the following commands:

• /sig <group-name>
• /wg <group-name>
• /committee <group-name>

Please see the group list for a listing of the SIGs, working groups, and committees available.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[k8s-ci-robot]

This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[neolit123]

a better place to ask would be on the support channels. please see:
https://git.k8s.io/kubernetes/SUPPORT.md

/kind support
/close

[k8s-ci-robot]

@neolit123: Closing this issue.

In response to this:

a better place to ask would be on the support channels. please see:
https://git.k8s.io/kubernetes/SUPPORT.md

/kind support
/close

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.