argo watch does not update/refresh screen

[ptemmer]

Checklist:

• I've included the version.
• I've included reproduction steps.
• I've included the workflow YAML.
• I've included the logs.

What happened:
argo watch WORKFLOW_NAME most of the time does not refresh it's output

What you expected to happen:
I expected argo watch to refresh it's output every few seconds as was the case with at least v2.4

How to reproduce it (as minimally and precisely as possible):
Submit a workflow and either add the '--watch' flag, or submit a workflow and execute the argo watch WORKFLOW_NAME afterwards.

Anything else we need to know?:

Environment:

• Argo version:

$ argo version
argo: v2.7.1
  BuildDate: 2020-04-07T16:40:24Z
  GitCommit: 2a3f59c10ae260a460b6ad97a0cadd8667d4b488
  GitTreeState: clean
  GitTag: v2.7.1
  GoVersion: go1.13.4
  Compiler: gc
  Platform: darwin/amd64

• Kubernetes version :

$ kubectl version -o yaml
clientVersion:
  buildDate: "2019-10-15T23:41:55Z"
  compiler: gc
  gitCommit: c97fe5036ef3df2967d086711e6c0c405941e14b
  gitTreeState: clean
  gitVersion: v1.16.2
  goVersion: go1.12.10
  major: "1"
  minor: "16"
  platform: darwin/amd64
serverVersion:
  buildDate: "2020-01-31T20:00:26Z"
  compiler: gc
  gitCommit: c52f59bbba5fbf21fbb18e9a06f96e563fe4c20a
  gitTreeState: clean
  gitVersion: v1.14.8
  goVersion: go1.12.10
  major: "1"
  minor: "14"
  platform: linux/amd64

---

Message from the maintainers:

If you are impacted by this bug please add a 👍 reaction to this issue! We often sort issues this way to know what to prioritize.

[alexec]

This maybe fixed in v2.7.2 CLI- can you please try that version?

[ptemmer]

Just tried v2.7.2 CLI against v2.7.1 in cluster.
The first 8 seconds the screen refreshed every 1-2 seconds, after which it stopped refreshing until after 38 seconds. It refreshed a few more times every 1-2 seconds to finally "freeze" and stop refreshing the last time 52 seconds in. The workflow was still running nonetheless.

Tomorrow I'll give it another shot also updating the workflow-controller to v2.7.2

[alexec]

Looking at the most recent code I notice it only updates when the workflow changed. So you should see a change every time a step changes state (e.g becomes complete) or the workflow changes state (e.g. completes). You should not expect it to just "update every few seconds" without a state change.

[ptemmer]

Well, initially that's the impression that I had. However, it definitely does not refresh after a while, although the step changed state.

I just tested with a workflow that finished little over 15min, however the argo watch command stopped refreshing after 1m6s when it was performing the 3rd step out of 11.

[alexec]

Can you please provide an example of the workflow so I can try and repro?

[alexec]

I have a potential fix, but I'm not greatly confident in it.

I forgot to ask can you run argo --loglevel DEBUG watch ... and see what that says?

[ptemmer]

I was able to reproduce with the following workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-nested-
spec:
  entrypoint: diamond
  templates:
  - name: echo
    inputs:
      parameters:
      - name: message
    container:
      image: alpine:3.7
      command: [sleep , 30]
  - name: diamond
    dag:
      tasks:
      - name: A
        template: nested-diamond
        arguments:
          parameters: [{name: message, value: A}]
      - name: B
        dependencies: [A]
        template: nested-diamond
        arguments:
          parameters: [{name: message, value: B}]
      - name: C
        dependencies: [A]
        template: nested-diamond
        arguments:
          parameters: [{name: message, value: C}]
      - name: D
        dependencies: [B, C]
        template: nested-diamond
        arguments:
          parameters: [{name: message, value: D}]
  - name: nested-diamond
    inputs:
      parameters:
      - name: message
    dag:
      tasks:
      - name: A
        template: echo
        arguments:
          parameters: [{name: message, value: "{{inputs.parameters.message}}A"}]
      - name: B
        dependencies: [A]
        template: echo
        arguments:
          parameters: [{name: message, value: "{{inputs.parameters.message}}B"}]
      - name: C
        dependencies: [A]
        template: echo
        arguments:
          parameters: [{name: message, value: "{{inputs.parameters.message}}C"}]
      - name: D
        dependencies: [B, C]
        template: echo
        arguments:
          parameters: [{name: message, value: "{{inputs.parameters.message}}D"}]

Below screenshot shows at the top the output of argo watch that "froze" after over a minute, while the bottom part shows the output of argo get after a few minutes.

[alexec]

Are you using argo server? Any kind of ingress?

Can I ask you to run with the --loglevel DEBUG command? Or did you already?

[ptemmer]

Yes, we are running argo-server in client auth mode.
We also have a few ingresses, however none that serve Argo (Argo-server is accessed through port-forward).

Please find the last lines of the debug output here

Namespace:           argo
ServiceAccount:      argo
Status:              Running
Created:             Thu Apr 16 23:45:03 +0200 (1 minute ago)
Started:             Thu Apr 16 23:45:03 +0200 (1 minute ago)
Duration:            1 minute 25 seconds

STEP                 TEMPLATE        PODNAME                      DURATION  MESSAGE
 ● dag-nested-jg6pz  diamond
 └-● A               nested-diamond
   ├-✔ A             echo            dag-nested-jg6pz-4063873041  46s
   ├-✔ B             echo            dag-nested-jg6pz-4013540184  35s
   ├-✔ C             echo            dag-nested-jg6pz-4030317803  34s
   └-◷ D             echo            dag-nested-jg6pz-4114205898  1s        ContainerCreating
DEBU[0095] Received event
DEBU[0095] Sending event                                 phase=Running type=MODIFIED workflow=dag-nested-jg6pz
Name:                dag-nested-jg6pz
Namespace:           argo
ServiceAccount:      argo
Status:              Running
Created:             Thu Apr 16 23:45:03 +0200 (1 minute ago)
Started:             Thu Apr 16 23:45:03 +0200 (1 minute ago)
Duration:            1 minute 29 seconds

STEP                 TEMPLATE        PODNAME                      DURATION  MESSAGE
 ● dag-nested-jg6pz  diamond
 └-● A               nested-diamond
   ├-✔ A             echo            dag-nested-jg6pz-4063873041  46s
   ├-✔ B             echo            dag-nested-jg6pz-4013540184  35s
   ├-✔ C             echo            dag-nested-jg6pz-4030317803  34s
   └-● D             echo            dag-nested-jg6pz-4114205898  5s
DEBU[0107] Result channel done