{{workflow.status}} is not working in metrics

[eyalyoli]

Checklist

• Double-checked my configuration.
• Tested using the latest version.
• Used the Emissary executor.

Summary

What happened/what you expected to happen?

Running this example workflow for custom metrics at the workflow & template levels on the latest version (v3.3.6) publishes the workflow templates without status label!

Workflow metrics (playground_workflow_duration, playground_workflow_result_counter) are visible in prometheus without the status label => {{workflow.status}} is not giving any value.
While template metrics (playground_workflow_duration_task_seconds, playground_workflow_result_task_counter) with status label of {{status}} is working fine.

What version are you running?

3.3.6

This previously worked on version 3.2.9

Diagnostics

Paste the smallest workflow that reproduces the bug. We must be able to run the workflow.

a shorter version of this example

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: dag-task-
spec:
  entrypoint: dag-task
  metrics: # Custom metric workflow level
    prometheus:
      - name: playground_workflow_new
        help: "Count of workflow execution by result status  - workflow level"
        labels:
          - key: "playground_id_workflow_counter"
            value: "test"
          - key: status
            value: "{{workflow.status}}"
        counter:
          value: "1"
  templates:
  - name: dag-task
    dag:
      tasks:
      - name: TEST-ONE
        template: echo

  - name: echo
    container:
      image: alpine:3.7
      command: [echo, "hello"]

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow} 

# If the workflow's pods have not been created, you can skip the rest of the diagnostics.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

@dpadhiar Can you take a look?