Unable to join nodes to cluster using microk8s join command

[bhargavsangabattula]

Hi All,
I am in process of setting of new microk8s cluster with 3 nodes which are running on Ubuntu 22.04 and kernel version is 6.5.0-1017-aws on AWS infra everytime I am getting the this error while joining the nodes to primary node to form the cluster and seeing this error when run the microk8s.status on secondary nodes.
ubuntu@ip-172-31-37-61:$ microk8s join 172.31.44.251:25000/f7159c7215fe49c70b71c79545b5c584/f496dd46d0b8
Contacting cluster at 172.31.44.251
Waiting for this node to finish joining the cluster. .. .. .. .. .. .. .. .. .. ..
ubuntu@ip-172-31-37-61:$ microk8s.status
microk8s is not running. Use microk8s inspect for a deeper inspection.
ubuntu@ip-172-31-37-61:~$ microk8s.inspect
Inspecting system
Inspecting Certificates
Inspecting services
Service snap.microk8s.daemon-cluster-agent is running
Service snap.microk8s.daemon-containerd is running
Service snap.microk8s.daemon-kubelite is running
Service snap.microk8s.daemon-k8s-dqlite is running
Service snap.microk8s.daemon-apiserver-kicker is running
Copy service arguments to the final report tarball
Inspecting AppArmor configuration
Gathering system information
Copy processes list to the final report tarball
Copy disk usage information to the final report tarball
Copy memory usage information to the final report tarball
Copy server uptime to the final report tarball
Copy openSSL information to the final report tarball
Copy snap list to the final report tarball
Copy VM name (or none) to the final report tarball
Copy current linux distribution to the final report tarball
Copy network configuration to the final report tarball
Inspecting kubernetes cluster
Inspect kubernetes cluster
Inspecting dqlite
Inspect dqlite
cp: cannot stat '/var/snap/microk8s/6673/var/kubernetes/backend/localnode.yaml': No such file or directory

Building the report tarball
Report tarball is at /var/snap/microk8s/6673/inspection-report-20240623_091714.tar.gz
ubuntu@ip-172-31-37-61:~$ cd /var/snap/microk8s/6673/
ubuntu@ip-172-31-37-61:/var/snap/microk8s/6673$ pwd
/var/snap/microk8s/6673
ubuntu@ip-172-31-37-61:/var/snap/microk8s/6673$ uname -t
uname: invalid option -- 't'
Try 'uname --help' for more information.
ubuntu@ip-172-31-37-61:/var/snap/microk8s/6673$ uname -r
6.5.0-1017-aws
ubuntu@ip-172-31-37-61:/var/snap/microk8s/6673$

and pls find my here microk8s.inspect tar file as well as i generated from secondary nodes.
inspection-report.zip

This is from primary node :

ubuntu@ip-172-31-44-251:$ microk8s kubectl get nodes
NAME STATUS ROLES AGE VERSION
ip-172-31-44-251 Ready 7m53s v1.26.15
ubuntu@ip-172-31-44-251:$ microk8s.status
microk8s is running
high-availability: no
datastore master nodes: 172.31.44.251:19001
datastore standby nodes: none
addons:
enabled:
dashboard # (core) The Kubernetes dashboard
dns # (core) CoreDNS
ha-cluster # (core) Configure high availability on the current node
helm # (core) Helm - the package manager for Kubernetes
helm3 # (core) Helm 3 - the package manager for Kubernetes
hostpath-storage # (core) Storage class; allocates storage from host directory
ingress # (core) Ingress controller for external access
metrics-server # (core) K8s Metrics Server for API access to service metrics
storage # (core) Alias to hostpath-storage add-on, deprecated
disabled:
cert-manager # (core) Cloud native certificate management
community # (core) The community addons repository
gpu # (core) Automatic enablement of Nvidia CUDA
host-access # (core) Allow Pods connecting to Host services smoothly
kube-ovn # (core) An advanced network fabric for Kubernetes
mayastor # (core) OpenEBS MayaStor
metallb # (core) Loadbalancer for your Kubernetes cluster
minio # (core) MinIO object storage
observability # (core) A lightweight observability stack for logs, traces and metrics
prometheus # (core) Prometheus operator for monitoring and logging
rbac # (core) Role-Based Access Control for authorisation
registry # (core) Private image registry exposed on localhost:32000
ubuntu@ip-172-31-44-251:$ uname -R
uname: invalid option -- 'R'
Try 'uname --help' for more information.
ubuntu@ip-172-31-44-251:$ uname -r
6.5.0-1017-aws
ubuntu@ip-172-31-44-251:~$

Please help to suggest or advise what is the problem here and why I am unable to join the nodes to form a cluster ?.

@ktsakalozos can you pls help me on this issue.