Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Pod argo-server keeps restarting on GCP #2249

Closed
4 tasks done
klaucos opened this issue Feb 17, 2020 · 8 comments · Fixed by #2253
Closed
4 tasks done

Pod argo-server keeps restarting on GCP #2249

klaucos opened this issue Feb 17, 2020 · 8 comments · Fixed by #2253
Labels
type/bug
Milestone
v2.5

Comments

@klaucos
Copy link

klaucos commented Feb 17, 2020 •
edited
Loading

Checklist:

  • I've included the version.
  • I've included reproduction steps.
  • I've included the workflow YAML. - Not applicable
  • I've included the logs.

What happened:
After update to Argo 1.5+ we noticed that newly introduced argo-server keeps restarting on Google Cloud platform(GCP).
What you expected to happen:
Do not restart in random intervals. I'd expect that server keeps running at all times.
How to reproduce it (as minimally and precisely as possible):
Deleted old argo namespace.
Recreate namespace and install argo on GCP:
kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/v2.5.0-rc12/manifests/install.yaml
Wait approx 30-40 minutes and pod with argo-server restarts.
It then keeps restarting approximately every 30-40 minutes.
Right before it restarts it logs message: 'config map event, exiting gracefully'

Anything else we need to know?:
Environment:

  • Argo version:
    Argo: v2.5.0-rc12
  • Kubernetes version :
clientVersion:
  buildDate: "2019-09-18T14:51:13Z"
  compiler: gc
  gitCommit: 67d2fcf276fcd9cf743ad4be9a9ef5828adc082f
  gitTreeState: clean
  gitVersion: v1.15.4
  goVersion: go1.12.9
  major: "1"
  minor: "15"
  platform: linux/amd64
serverVersion:
  buildDate: "2020-02-07T00:50:57Z"
  compiler: gc
  gitCommit: a9973cbb2722793e2ea08d20880633ca61d3e669
  gitTreeState: clean
  gitVersion: v1.15.9-gke.8
  goVersion: go1.12.12b4
  major: "1"
  minor: 15+
  platform: linux/amd64

Other debugging information (if applicable):

  • argo-server logs:
time="2020-02-17T11:40:48Z" level=info authMode=server baseHRef=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/ managedNamespace= namespace=argo
time="2020-02-17T11:40:48Z" level=warning msg="ConfigMap 'workflow-controller-configmap' does not have key 'config'"
time="2020-02-17T11:40:48Z" level=info msg="Config Map: "
time="2020-02-17T11:40:48Z" level=info msg="Argo Server started successfully on address :2746"
time="2020-02-17T11:45:48Z" level=info msg="Alloc=5522 TotalAlloc=13159 Sys=70590 NumGC=7 Goroutines=52"
time="2020-02-17T11:50:48Z" level=info msg="Alloc=5677 TotalAlloc=14384 Sys=70590 NumGC=9 Goroutines=52"
time="2020-02-17T11:55:48Z" level=info msg="Alloc=5437 TotalAlloc=15621 Sys=70590 NumGC=12 Goroutines=52"
time="2020-02-17T12:00:48Z" level=info msg="Alloc=5670 TotalAlloc=16839 Sys=70590 NumGC=14 Goroutines=52"
time="2020-02-17T12:05:48Z" level=info msg="Alloc=5354 TotalAlloc=18084 Sys=70590 NumGC=17 Goroutines=52"
time="2020-02-17T12:10:48Z" level=info msg="Alloc=5605 TotalAlloc=19307 Sys=70590 NumGC=19 Goroutines=52"
time="2020-02-17T12:15:48Z" level=info msg="Alloc=5264 TotalAlloc=20513 Sys=70590 NumGC=22 Goroutines=52"
time="2020-02-17T12:20:48Z" level=info msg="Alloc=5519 TotalAlloc=21751 Sys=70590 NumGC=24 Goroutines=52"
time="2020-02-17T12:25:48Z" level=info msg="Alloc=5761 TotalAlloc=22974 Sys=70590 NumGC=27 Goroutines=52"
time="2020-02-17T12:26:36Z" level=info msg="config map event, exiting gracefully" eventType=

After that it restarts.

  • workflow-controller logs:
time="2020-02-17T11:25:16Z" level=info msg="Updated phase  -> Running" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:16Z" level=info msg="Pod node {hello-world-2qmh7 hello-world-2qmh7 hello-world-2qmh7 Pod whalesay nil    Pending   2020-02-17 11:25:16.339949808 +0000 UTC 0001-01-01 00:00:00 +0000 UTC  <nil> nil nil [] []} initialized Pending" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:16Z" level=info msg="Created pod: hello-world-2qmh7 (hello-world-2qmh7)" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:16Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=148795105 workflow=hello-world-2qmh7
time="2020-02-17T11:25:17Z" level=info msg="Processing workflow" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:17Z" level=info msg="Updating node &NodeStatus{ID:hello-world-2qmh7,Name:hello-world-2qmh7,DisplayName:hello-world-2qmh7,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Pending,BoundaryID:,Message:,StartedAt:2020-02-17 11:25:16 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,} message: ContainerCreating"
time="2020-02-17T11:25:17Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=148795111 workflow=hello-world-2qmh7
time="2020-02-17T11:25:18Z" level=info msg="Processing workflow" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:25Z" level=info msg="Processing workflow" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:25Z" level=info msg="Updating node &NodeStatus{ID:hello-world-2qmh7,Name:hello-world-2qmh7,DisplayName:hello-world-2qmh7,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Pending,BoundaryID:,Message:ContainerCreating,StartedAt:2020-02-17 11:25:16 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,} status Pending -> Running"
time="2020-02-17T11:25:25Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=148795152 workflow=hello-world-2qmh7
time="2020-02-17T11:25:26Z" level=info msg="Processing workflow" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:26Z" level=info msg="Processing workflow" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:26Z" level=info msg="Updating node &NodeStatus{ID:hello-world-2qmh7,Name:hello-world-2qmh7,DisplayName:hello-world-2qmh7,Type:Pod,TemplateName:whalesay,TemplateRef:nil,Phase:Running,BoundaryID:,Message:,StartedAt:2020-02-17 11:25:16 +0000 UTC,FinishedAt:0001-01-01 00:00:00 +0000 UTC,PodIP:,Daemoned:nil,Inputs:nil,Outputs:nil,Children:[],OutboundNodes:[],StoredTemplateID:,WorkflowTemplateName:,TemplateScope:,} status Running -> Succeeded"
time="2020-02-17T11:25:26Z" level=info msg="Updated phase Running -> Succeeded" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:26Z" level=info msg="Marking workflow completed" namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:26Z" level=info msg="Checking daemoned children of " namespace=default workflow=hello-world-2qmh7
time="2020-02-17T11:25:26Z" level=info msg="Workflow update successful" namespace=default phase=Succeeded resourceVersion=148795163 workflow=hello-world-2qmh7
time="2020-02-17T11:25:27Z" level=info msg="Labeled pod default/wf-name
W0217 11:26:58.385829       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148793597 (148795163)
time="2020-02-17T11:27:23Z" level=info msg="Alloc=6226 TotalAlloc=30589 Sys=70590 NumGC=18 Goroutines=102"
time="2020-02-17T11:27:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T11:27:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T11:27:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T11:27:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T11:27:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T11:27:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T11:27:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T11:27:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T11:27:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T11:27:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T11:27:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T11:27:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T11:32:23Z" level=info msg="Alloc=6185 TotalAlloc=32250 Sys=70590 NumGC=20 Goroutines=102"
W0217 11:36:53.394892       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148795583 (148797924)
time="2020-02-17T11:37:23Z" level=info msg="Alloc=6177 TotalAlloc=32444 Sys=70590 NumGC=23 Goroutines=102"
W0217 11:42:01.412026       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148798339 (148799315)
time="2020-02-17T11:42:23Z" level=info msg="Alloc=6244 TotalAlloc=32546 Sys=70590 NumGC=25 Goroutines=102"
time="2020-02-17T11:47:23Z" level=info msg="Alloc=6179 TotalAlloc=32645 Sys=70590 NumGC=28 Goroutines=102"
time="2020-02-17T11:47:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T11:47:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T11:47:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T11:47:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T11:47:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T11:47:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T11:47:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T11:47:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T11:47:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T11:47:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T11:47:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T11:47:23Z" level=info msg="Enforcing history limit for wf-name"
W0217 11:51:16.422035       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148799745 (148801900)
time="2020-02-17T11:52:23Z" level=info msg="Alloc=6197 TotalAlloc=34427 Sys=70590 NumGC=30 Goroutines=102"
time="2020-02-17T11:57:23Z" level=info msg="Alloc=6180 TotalAlloc=34522 Sys=70590 NumGC=33 Goroutines=102"
W0217 12:00:45.429689       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148802318 (148804541)
time="2020-02-17T12:02:23Z" level=info msg="Alloc=6181 TotalAlloc=34666 Sys=70590 NumGC=35 Goroutines=102"
time="2020-02-17T12:07:23Z" level=info msg="Alloc=6182 TotalAlloc=34797 Sys=70590 NumGC=38 Goroutines=102"
time="2020-02-17T12:07:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T12:07:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T12:07:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T12:07:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T12:07:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T12:07:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T12:07:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T12:07:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T12:07:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T12:07:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T12:07:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T12:07:23Z" level=info msg="Enforcing history limit for wf-name"
W0217 12:09:55.442355       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148804977 (148807110)
time="2020-02-17T12:12:23Z" level=info msg="Alloc=6202 TotalAlloc=36487 Sys=70590 NumGC=40 Goroutines=102"
W0217 12:16:49.455300       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148807526 (148809042)
time="2020-02-17T12:17:23Z" level=info msg="Alloc=6284 TotalAlloc=36649 Sys=70590 NumGC=42 Goroutines=102"
time="2020-02-17T12:22:23Z" level=info msg="Alloc=6204 TotalAlloc=36728 Sys=70590 NumGC=45 Goroutines=102"
W0217 12:26:12.463837       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148809456 (148811649)
time="2020-02-17T12:27:23Z" level=info msg="Alloc=6280 TotalAlloc=36883 Sys=70590 NumGC=47 Goroutines=102"
time="2020-02-17T12:27:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T12:27:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T12:27:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T12:27:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T12:27:23Z" level=info msg="Processing default/wf-name
time="2020-02-17T12:27:23Z" level=info msg="CronWorkflow default/wf-name
time="2020-02-17T12:27:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T12:27:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T12:27:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T12:27:23Z" level=info msg="Enforcing history limit for wf-name"
time="2020-02-17T12:27:23Z" level=warning msg="Workflow wf-name completed"
time="2020-02-17T12:27:23Z" level=info msg="Enforcing history limit for wf-name"
W0217 12:31:39.476293       1 reflector.go:302] github.com/argoproj/argo/workflow/controller/config_controller.go:147: watch of *v1.ConfigMap ended with: too old resource version: 148812068 (148813140)
time="2020-02-17T12:32:23Z" level=info msg="Alloc=6251 TotalAlloc=38611 Sys=70590 NumGC=50 Goroutines=102"

Installation logs:

kubectl apply -n argo -f https://raw.githubusercontent.com/argoproj/argo/v2.5.0-rc12/manifests/install.yaml
customresourcedefinition.apiextensions.k8s.io/cronworkflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflows.argoproj.io unchanged
customresourcedefinition.apiextensions.k8s.io/workflowtemplates.argoproj.io unchanged
serviceaccount/argo created
serviceaccount/argo-server created
role.rbac.authorization.k8s.io/argo-role created
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-admin unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-edit unchanged
clusterrole.rbac.authorization.k8s.io/argo-aggregate-to-view unchanged
clusterrole.rbac.authorization.k8s.io/argo-cluster-role unchanged
clusterrole.rbac.authorization.k8s.io/argo-server-cluster-role unchanged
rolebinding.rbac.authorization.k8s.io/argo-binding created
clusterrolebinding.rbac.authorization.k8s.io/argo-binding unchanged
clusterrolebinding.rbac.authorization.k8s.io/argo-server-binding unchanged
configmap/workflow-controller-configmap created
service/argo-server created
deployment.apps/argo-server created
deployment.apps/workflow-controller created

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.
@sarabala1979
Copy link
Member

I think your workflow-controller-configmap is getting updated a few times. Argo server has been implemented to restart gracefully to get the updated configmap. This is by design. It may be a small-time interruption on UI or API calls. Instead of reloading configmap and handle the changes, @alexec thought this is the easiest and safest way. Do you think the restart causing any other issues?

@alexec alexec added this to the v2.5 milestone Feb 17, 2020
@alexec
Copy link
Contributor

alexec commented Feb 17, 2020

Argo Server exits whenever the config map changes:

https://github.com/argoproj/argo/blob/79f13373fd8c4d0e9c9ff56f2133fa6009d1ed07/server/apiserver/argoserver.go#L283

It is possible you are getting some other event (deletion? bookmark?). Can you please attach a list of events on the workflow-controller-configmapat that time?

@klaucos
Copy link
Author

klaucos commented Feb 18, 2020

I did not notice any issues that would affect functionality.
We monitor all restarts of services across our clusters and it raised a red flag that Argo-server kept restarting.
Its slightly unexpected behavior but we can whitelist argo-server and ignore restarts in the future.

It is restarting approx every 40 mins.

NAME                                  READY   STATUS    RESTARTS   AGE
argo-server-589fb4b754-rn5bq          1/1     Running   29         21h
workflow-controller-65ccd954c-jk7rs   1/1     Running   0          21h

I do not know which events are present at the time of restarts:
kubectl describe configmap/workflow-controller-configmap -n argo

Name:         workflow-controller-configmap
Namespace:    argo
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"v1","kind":"ConfigMap","metadata":{"annotations":{},"name":"workflow-controller-configmap","namespace":"argo"}}

Data
====
Events:  <none>

kubectl get events -n argo

LAST SEEN   TYPE     REASON    OBJECT                             MESSAGE
40m         Normal   Created   pod/argo-server-589fb4b754-rn5bq   Created container argo-server
40m         Normal   Started   pod/argo-server-589fb4b754-rn5bq   Started container argo-server
40m         Normal   Pulled    pod/argo-server-589fb4b754-rn5bq   Container image "argoproj/argocli:v2.5.0-rc12" already present on machine

alexec added a commit to alexec/argo-workflows that referenced this issue Feb 18, 2020
@alexec
fix: Ignore bookmark events for restart. Fixes argoproj#2249
2f73779
@alexec alexec mentioned this issue Feb 18, 2020
Merged
5 tasks
alexec added a commit that referenced this issue Feb 18, 2020
@alexec
fix: Ignore bookmark events for restart. Fixes #2249 (#2253)
8c97689
alexec added a commit that referenced this issue Feb 18, 2020
@alexec
fix: Ignore bookmark events for restart. Fixes #2249 (#2253)
661f8a1
@descrepes
Copy link
Contributor

Hi,
We have exactly the same behavior on Azure.
We are running Argo v2.6.0 and Kubernetes v1.14.8 on AKS.

time="2020-03-02T14:49:58Z" level=info msg="Creating DB session"
time="2020-03-02T14:49:58Z" level=info msg="Node status offloading config" ttl=5m0s
time="2020-03-02T14:49:58Z" level=info msg="Argo Server started successfully on address :2746"
time="2020-03-02T14:54:58Z" level=info msg="Alloc=4065 TotalAlloc=6992 Sys=70592 NumGC=4 Goroutines=53"
time="2020-03-02T14:59:58Z" level=info msg="Alloc=3835 TotalAlloc=8185 Sys=70592 NumGC=7 Goroutines=53"
time="2020-03-02T15:04:58Z" level=info msg="Alloc=3988 TotalAlloc=9345 Sys=70592 NumGC=9 Goroutines=53"
time="2020-03-02T15:09:58Z" level=info msg="Alloc=3745 TotalAlloc=10540 Sys=70592 NumGC=12 Goroutines=53"
time="2020-03-02T15:14:58Z" level=info msg="Alloc=3920 TotalAlloc=11740 Sys=70592 NumGC=14 Goroutines=53"
time="2020-03-02T15:19:58Z" level=info msg="Alloc=3676 TotalAlloc=12929 Sys=70592 NumGC=17 Goroutines=53"
time="2020-03-02T15:24:58Z" level=info msg="Alloc=3833 TotalAlloc=14136 Sys=70592 NumGC=19 Goroutines=53"
time="2020-03-02T15:29:58Z" level=info msg="Alloc=4064 TotalAlloc=15307 Sys=70592 NumGC=21 Goroutines=53"
time="2020-03-02T15:34:58Z" level=info msg="Alloc=3758 TotalAlloc=16502 Sys=70592 NumGC=24 Goroutines=53"
time="2020-03-02T15:39:58Z" level=info msg="Alloc=3999 TotalAlloc=17673 Sys=70592 NumGC=26 Goroutines=53"
time="2020-03-02T15:44:58Z" level=info msg="Alloc=3678 TotalAlloc=18865 Sys=70592 NumGC=29 Goroutines=53"
time="2020-03-02T15:48:41Z" level=info msg="config map event, exiting gracefully" eventType=

I can't find the event type that trigger this restart.
Let me know if you need more logs.

Regards,

Alexandre

@dmegyesi
Copy link

dmegyesi commented Mar 4, 2020

Hi,

+1, we have the same behaviour on AWS. Running Argo v2.6.0 on EKS v1.14.9:

time="2020-03-04T08:24:09Z" level=warning msg="ConfigMap 'workflow-controller-configmap' does not have key 'config'"
time="2020-03-04T08:24:09Z" level=info msg="Config Map: "
time="2020-03-04T08:24:09Z" level=info msg="Argo Server started successfully on address :2746"
time="2020-03-04T08:29:09Z" level=info msg="Alloc=4035 TotalAlloc=6847 Sys=70080 NumGC=3 Goroutines=52"
time="2020-03-04T08:34:09Z" level=info msg="Alloc=3720 TotalAlloc=8484 Sys=70080 NumGC=6 Goroutines=52"
time="2020-03-04T08:39:09Z" level=info msg="Alloc=3959 TotalAlloc=10056 Sys=70080 NumGC=8 Goroutines=52"
time="2020-03-04T08:44:09Z" level=info msg="Alloc=3568 TotalAlloc=11702 Sys=70080 NumGC=11 Goroutines=52"
time="2020-03-04T08:49:09Z" level=info msg="Alloc=3885 TotalAlloc=13343 Sys=70080 NumGC=13 Goroutines=52"
time="2020-03-04T08:54:09Z" level=info msg="Alloc=3839 TotalAlloc=14977 Sys=70080 NumGC=16 Goroutines=52"
time="2020-03-04T08:59:09Z" level=info msg="Alloc=3808 TotalAlloc=16534 Sys=70080 NumGC=18 Goroutines=52"
time="2020-03-04T09:04:09Z" level=info msg="Alloc=4116 TotalAlloc=18167 Sys=70080 NumGC=20 Goroutines=52"
time="2020-03-04T09:09:09Z" level=info msg="Alloc=3723 TotalAlloc=19793 Sys=70080 NumGC=23 Goroutines=52"
time="2020-03-04T09:14:09Z" level=info msg="Alloc=4040 TotalAlloc=21426 Sys=70080 NumGC=25 Goroutines=52"
time="2020-03-04T09:19:09Z" level=info msg="Alloc=3568 TotalAlloc=22991 Sys=70080 NumGC=28 Goroutines=52"
time="2020-03-04T09:23:56Z" level=info msg="config map event, exiting gracefully" eventType=

The ConfigMap is 0 bytes, there is no config in it. Let me know if we can supply any additional info that could be useful.

@dmegyesi dmegyesi mentioned this issue Mar 4, 2020
Closed
4 tasks
@alexec
Copy link
Contributor

alexec commented Mar 4, 2020

Interesting. Can you please create any empty config in the map and see what happens?

@descrepes
Copy link
Contributor

The 2.6.1 release fixed the issue.
Thank you so much for your work !

@simster7
Copy link
Member

Fully fixed with #2360

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
type/bug
Projects
None yet
Milestone
v2.5
Development

Successfully merging a pull request may close this issue.

fix: Ignore bookmark events for restart. Fixes #2249 alexec/argo-workflows
6 participants
@simster7 @alexec @dmegyesi @descrepes @sarabala1979 @klaucos