valueFrom configMap not working in workflow level parameters

[liuzqt]

Summary

What happened/what you expected to happen?

reproducing template:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: my-pipeline
spec:
  entrypoint: main
  arguments:
    parameters: 
      - name: myvalue
        valueFrom:
          configMapKeyRef:
            name: my-configuration
            key: myvalue
  templates:
    - name: main
      dag:
        tasks:
          - name: dummy
            inline:
              container:
                image: docker/whalesay:latest
                command: [sh, -c]
                args: ["echo {{workflow.parameters.myvalue}}"]

workflow spec:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: my-
spec:
  arguments: {}
  workflowTemplateRef:
    name: my-pipeline

as you can see, I declared the workflow level arguments myvalue, and the value is from configMap, but this is not working the value is empty.

I'm sure the config map resource has been created successfully, and moreover, if I declare the parameters in the dag instead of the workflow level arguments, it's working well, something like:

  templates:
    - name: main
      inputs:
      	parameters:
	      - name: myvalue
		    valueFrom:
		      configMapKeyRef:
		        name: my-configuration
		        key: myvalue
      dag:
      ...

What version of Argo Workflows are you running?

3.2.2

Diagnostics

reproducing workflow template is given above

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[terrytangyuan]

Could you provide any error messages, controller logs, etc. as mentioned in the issue template?

[sarabala1979]

@liuzqt Is it failing in CLI during submission?

[liuzqt]

@sarabala1979 @terrytangyuan It didn't fail during submission, and neither fail during runtime. Argo itself is running well, except the input value is empty(not the expected value from config map), and the error is within my own application due to the empty value.

[terrytangyuan]

Could you provide controller log and raw YAML of your configmap?

Also, does this example work for you? https://github.com/argoproj/argo-workflows/blob/master/examples/global-parameters-from-configmap.yaml

[liuzqt]

@terrytangyuan yes that example works well.
After digging a while, I got a more detailed reproducing example:

Actually, the way I do it is that I create a Workflow template in the cluster, and then submit a workflow referencing that template, as documented here(https://argoproj.github.io/argo-workflows/workflow-templates/), instead of directly create a workflow.

The WorkflowTemplate:

apiVersion: argoproj.io/v1alpha1
kind: WorkflowTemplate
metadata:
  name: patch-processing-pipeline
spec:
  entrypoint: runner1
  arguments:
    parameters:
      - name: s3-bucket
        valueFrom:
          configMapKeyRef:
            name: patch-processing-configuration
            key: s3-bucket
      - name: s3-prefix
        valueFrom:
          configMapKeyRef:
            name: patch-processing-configuration
            key: s3-prefix
      - name: param1
  templates:
    - name: runner1
      inputs:
        parameters:
          - name: global-s3-prefix
            value: "{{workflow.parameters.s3-prefix}}"
          - name: global-s3-bucket
            value: "{{workflow.parameters.s3-bucket}}"
          - name: global-param1
            value: "{{workflow.parameters.param1}}"
      container:
        image: docker/whalesay:latest
        command: [sh, -c]
        args: ["echo this is runner1 {{workflow.parameters.s3-bucket}} {{workflow.parameters.s3-prefix}} {{workflow.parameters.param1}}"]

The workflow:

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: patch-processing-
spec:
  arguments:
    parameters:
      - name: param1
        value: "2021-11-10-13-53-35"
  workflowTemplateRef:
    name: patch-processing-pipeline

And the result looks like an undefined behavior to me:
https://ibb.co/9N899LW

as you can see in the picture url, the param1 somehow overwrites the s3-bucket param. And my previous symptom of seeing empty value is possibly overwritten by some other parameters.

I suspect this is related to "store the template in the cluster and submit a workflow referencing it", because this issue is gone if I directly submit the workflow.

[liuzqt]

@terrytangyuan the raw config map:

apiVersion: v1
kind: ConfigMap
metadata:
  name: patch-processing-configuration
  labels:
    # Note that this label is required for the informer to detect this ConfigMap.
    workflows.argoproj.io/configmap-type: Parameter
data:
  # some other configurations
  # s3 configuration
  s3-bucket: {{ .Values.s3_bucket}}
  s3-prefix: {{ .Values.s3_prefix}}

As for the controller log, I might need to ask our Argo SRE......I'm running jobs on companies' cluster, will also attach here once I get the log

[liuzqt]

One more finding: if I put params from ConfigMap at the end of the param list, the issue is just gone. I will take this approach as a temporary fix.

[terrytangyuan]

Yes, this looks like a bug -- somehow the positioning affects the values. Note that if you want to use it as a workaround, you'd want to put an unused parameter as the first item in the list since the first parameter value is incorrect.

[terrytangyuan]

Something like the following:

  arguments:
    parameters:
      - name: unused
        value: "unused"
      - name: s3-prefix
        valueFrom:
          configMapKeyRef:
            name: patch-processing-configuration
            key: s3-prefix
      - name: s3-bucket
        valueFrom:
          configMapKeyRef:
            name: patch-processing-configuration
            key: s3-bucket

[liuzqt]

@terrytangyuan thanks, will use it as a workaround

[liuzqt]

Hi @terrytangyuan do we have plans to include this fix in the coming release?

[terrytangyuan]

Yes it will be included but no timeline yet

[liuzqt]

@terrytangyuan thanks

[diazGT94]

Hello I have a question, how can you access to the raw config map?

[thienmai1325]

Hi @terrytangyuan,

https://github.com/argoproj/argo-workflows/blob/master/examples/global-parameters-from-configmap.yaml

This example doesn't work to me, it's empty value ("") when I call it as global parameters.

Is this a bugs?