How do you properly restart the mircoshift machine #172

tosin2013 · 2021-07-21T18:24:33Z

tosin2013
Jul 21, 2021

Is there any recommendations on restart the machine that has microshift?
Does the service automatically restart microshift?

I restarted my box and received the error below.

$ systemctl status microshift.service                                                                                                                                                                                                                
× microshift.service - Microshift
     Loaded: loaded (/usr/lib/systemd/system/microshift.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Wed 2021-07-21 13:30:31 EDT; 53min ago
   Main PID: 876 (code=exited, status=1/FAILURE)
        CPU: 473ms

Jul 21 13:30:31 fedora systemd[1]: microshift.service: Scheduled restart job, restart counter is at 5.
Jul 21 13:30:31 fedora systemd[1]: Stopped Microshift.
Jul 21 13:30:31 fedora systemd[1]: microshift.service: Start request repeated too quickly.
Jul 21 13:30:31 fedora systemd[1]: microshift.service: Failed with result 'exit-code'.
Jul 21 13:30:31 fedora systemd[1]: Failed to start Microshift.

$ cat /etc/redhat-release
Fedora release 34 (Thirty Four)

rootfs · 2021-07-21T19:08:50Z

rootfs
Jul 21, 2021

can you provide journalctl -xe output?

1 reply

rootfs Jul 21, 2021

cc @oglok @copejon @cooktheryan

cooktheryan · 2021-07-21T19:38:53Z

cooktheryan
Jul 21, 2021

On RHEL I just issued a reboot and everything came back up as expected. This may be something specifically fedora related

[root@ip-172-31-68-145 ~]# kubectl run nginx --image nginx
pod/nginx created
[root@ip-172-31-68-145 ~]# oc get po -w
-bash: oc: command not found
[root@ip-172-31-68-145 ~]# kubectl get po -w
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          6s
^C[root@ip-172-31-68-145 ~]# reboot
Connection to 3.231.220.57 closed by remote host.
Connection to 3.231.220.57 closed.
rcook (test/api-ci-ln-rdb8x3t-f76d1-origin-ci-int-gce-dev-openshift-com:6443/system:admin)  @ ~/git/microshift
rhel-issues└─ $ ssh [email protected]
Last login: Wed Jul 21 19:16:23 2021 from 99.45.71.93
[ec2-user@ip-172-31-68-145 ~]$ sudo su -
Last login: Wed Jul 21 19:16:36 UTC 2021 on pts/0
[root@ip-172-31-68-145 ~]# kubectl get po 
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          4m17s

0 replies

cooktheryan · 2021-07-21T19:51:18Z

cooktheryan
Jul 21, 2021

Didn't run into any issues with a new fedora instance

[root@ip-172-31-71-113 ~]# kubectl get po 
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          31s
[root@ip-172-31-71-113 ~]# reboot
Connection to 34.239.154.230 closed by remote host.
Connection to 34.239.154.230 closed.
rcook (test/api-ci-ln-rdb8x3t-f76d1-origin-ci-int-gce-dev-openshift-com:6443/system:admin)  @ ~/git/microshift
rhel-issues└─ $ ssh [email protected]
Last login: Wed Jul 21 19:42:22 2021 from 99.45.71.93
[fedora@ip-172-31-71-113 ~]$ sudo su -
Last login: Wed Jul 21 19:42:24 UTC 2021 on pts/0
[root@ip-172-31-71-113 ~]# kubectl get po
NAME    READY   STATUS    RESTARTS   AGE
nginx   1/1     Running   0          2m33s
[root@ip-172-31-71-113 ~]# cat /etc/fedora-release
Fedora release 34 (Thirty Four)

1 reply

tosin2013 Jul 21, 2021
Author

I was running everything as a sudo user.

tosin2013 · 2021-07-21T20:20:45Z

tosin2013
Jul 21, 2021
Author

I attempted to install istio on the cluster and it looks like it is crashing the microshift deployment on restart. When it attempting to get the cpu resource metric.

I was trying to install istio-1.10.3

-- Journal begins at Wed 2021-07-21 12:31:16 EDT, ends at Wed 2021-07-21 16:17:29 EDT. --
-- Journal begins at Wed 2021-07-21 12:31:16 EDT, ends at Wed 2021-07-21 16:17:29 EDT. --
Jul 21 16:12:48 fedora microshift[944]: I0721 16:12:48.133828     944 event.go:291] "Event occurred" object="istio-system/istiod" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2beta2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cpu ut>
Jul 21 16:12:48 fedora microshift[944]: I0721 16:12:48.133917     944 event.go:291] "Event occurred" object="istio-system/istiod" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2beta2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid metrics>
Jul 21 16:12:48 fedora microshift[944]: E0721 16:12:48.149944     944 horizontal.go:227] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/activator: invalid metrics (1 invalid out of 1), first error is: failed to get cpu>
Jul 21 16:12:48 fedora microshift[944]: I0721 16:12:48.150741     944 event.go:291] "Event occurred" object="knative-serving/activator" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2beta2" type="Warning" reason="FailedGetResourceMetric" message="failed to get >
Jul 21 16:12:48 fedora microshift[944]: I0721 16:12:48.150840     944 event.go:291] "Event occurred" object="knative-serving/activator" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2beta2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid m>
Jul 21 16:12:48 fedora microshift[944]: E0721 16:12:48.157789     944 horizontal.go:227] failed to compute desired number of replicas based on listed metrics for Deployment/knative-serving/webhook: invalid metrics (1 invalid out of 1), first error is: failed to get cpu u>
Jul 21 16:12:48 fedora microshift[944]: I0721 16:12:48.157978     944 event.go:291] "Event occurred" object="knative-serving/webhook" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2beta2" type="Warning" reason="FailedGetResourceMetric" message="failed to get cp>
Jul 21 16:12:48 fedora microshift[944]: I0721 16:12:48.158060     944 event.go:291] "Event occurred" object="knative-serving/webhook" kind="HorizontalPodAutoscaler" apiVersion="autoscaling/v2beta2" type="Warning" reason="FailedComputeMetricsReplicas" message="invalid met>
Jul 21 16:12:48 fedora microshift[944]: {"level":"warn","ts":"2021-07-21T16:12:48.307-0400","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"144.617917ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/even>
Jul 21 16:12:48 fedora microshift[944]: {"level":"info","ts":"2021-07-21T16:12:48.307-0400","caller":"traceutil/trace.go:145","msg":"trace[620089184] range","detail":"{range_begin:/registry/events/istio-system/istiod.1693e767a773fc38; range_end:; response_count:1; respon>
Jul 21 16:12:48 fedora microshift[944]: {"level":"warn","ts":"2021-07-21T16:12:48.549-0400","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"198.767186ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/pods>
Jul 21 16:12:48 fedora microshift[944]: {"level":"info","ts":"2021-07-21T16:12:48.550-0400","caller":"traceutil/trace.go:145","msg":"trace[21190676] range","detail":"{range_begin:/registry/pods/olm/packageserver-8848f6957-cwkrs; range_end:; response_count:1; response_rev>
Jul 21 16:12:48 fedora microshift[944]: {"level":"warn","ts":"2021-07-21T16:12:48.920-0400","caller":"etcdserver/util.go:163","msg":"apply request took too long","took":"359.244258ms","expected-duration":"100ms","prefix":"read-only range ","request":"key:\"/registry/even>
Jul 21 16:12:48 fedora microshift[944]: {"level":"info","ts":"2021-07-21T16:12:48.921-0400","caller":"traceutil/trace.go:145","msg":"trace[1570538332] range","detail":"{range_begin:/registry/events/knative-serving/webhook.1693e767a83d4626; range_end:; response_count:1; r>

1 reply

mangelajo Feb 16, 2022
Collaborator

Probably Istio relies on lots of metrics that MicroShift is not exposing.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How do you properly restart the mircoshift machine #172

{{title}}

Replies: 4 comments 3 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

{{title}}

{{editor}}'s edit

{{editor}}'s edit

{{title}}

Select a reply

How do you properly restart the mircoshift machine #172

tosin2013 Jul 21, 2021

Replies: 4 comments · 3 replies

rootfs Jul 21, 2021

rootfs Jul 21, 2021

cooktheryan Jul 21, 2021

cooktheryan Jul 21, 2021

tosin2013 Jul 21, 2021 Author

tosin2013 Jul 21, 2021 Author

mangelajo Feb 16, 2022 Collaborator

tosin2013
Jul 21, 2021

Replies: 4 comments 3 replies

rootfs
Jul 21, 2021

cooktheryan
Jul 21, 2021

cooktheryan
Jul 21, 2021

tosin2013 Jul 21, 2021
Author

tosin2013
Jul 21, 2021
Author

mangelajo Feb 16, 2022
Collaborator