argo fails to list workflows when there is a big workflow in the list (more then ~30,000 steps/nodes)

[dpeer6]

Summary

When there is a big workflow, in any state, "argo list" fails with a similar message:
rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (238029029 vs. 104857600)

We are running many workflows, some of the big, with many steps/pods running, but when there is even only one with more then 25k+ items in the progress status, argo fails to list current workflows in the namespace... sometime the UI also fails to list it.

I can run the following command on other workflows, but not on the big one:
"argo get/delete/terminate {workflow}"

It does not happen when I use kubectl to get the current workflows, for example:
"kubectl get workflow -n argo"

I can also get the workflow progress status, running:
"kubectl get workflow {workflow} -n argo -o json | jq .status.progress"

General info:
We do use podGC policy, so completed pods and deleted.
We do archive and offload immediately completed workflows to a Postgre DB.
We do delete workflows every few hours so the workflow list kept short.

We use S3 to save Argo workflows logs and Artifacts.

What happened/what you expected to happen?
I would expect that argo cli won't fail to list argo workflows
I would expect that argo cli won't fail to get a big argo workflow, supporting paging or some other method

We are so happy with Argo and would like to understand if the message size limit can be adjusted/changed somehow, to support huge workflows.

Diagnostics

👀 Yes! We need all of your diagnostics, please make sure you add it all, otherwise we'll go around in circles asking you for it:

What Kubernetes provider are you using?
EKS 1.18

What version of Argo Workflows are you running?
3.0.7

What executor are you running? Docker/K8SAPI/Kubelet/PNS/Emissary
Docker

Did this work in a previous version? I.e. is it a regression?
N/A

Are you pasting thousands of log lines? That's too much information.

# Either a workflow that reproduces the bug, or paste you whole workflow YAML, including status, something like:
kubectl get wf -o yaml ${workflow}

# Logs from the workflow controller:
kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Sorry, but the log is huge and contain only info messages that all looks like normal and proper operation.

# The workflow's pods that are problematic:
kubectl get pod -o yaml -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

# Logs from in your workflow's wait container, something like:
kubectl logs -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.

[sarabala1979]

Argo API Server configured 104857600 as max size. we can parameterize it as an environment variable in argo-server. Would you interest to contribute this enhancement?

argo-workflows/server/apiserver/argoserver.go

Line 62 in 0e94283

MaxGRPCMessageSize = 100 * 1024 * 1024

[alexec]

We could also consider stripping managedFields from the API as per #5778 .

[alexec]

Oh - I think I made a performance improvement to the UI for something similar: #6004 which was ported to v3.1

[alexec]

@dpeer6 can you please test in v3.1?

[dpeer6]

Hi Guys,

Thanks for your time.

I have few questions:

1. Yes, I can test it with 3.1 but, UI fixes does not fix the argo cli issue, right?
2. If I build argo server with larger message size, will the cli/ui can handle it? will it support paginated data fetching or something similar, properly?
3. Yes, I would love to contribute the code to support larger message size, but I would love to implement it as an argument to argo server on startup time (using ENV variable or something similar). Not sure it is possible though currently, without massive change given the fact that I need to learn go :-)

[alexec]

Let me know how your testing goes!

[dpeer6]

Hi,

Our testings with 3.1.1 showed improvement with the UI, it did not fail/crash or timeout when we run 4-5 workflows, at the same time, with 20k nodes/steps/pods each, but the cli did returned with the same issue, when running: "argo list"
FATA[2021-06-29T12:07:24.513Z] rpc error: code = ResourceExhausted desc = grpc: trying to send message larger than max (214553096 vs. 104857600)

and at some point later I also got this reply using the CLI: ("argo list")
FATA[2021-06-29T12:19:21.808Z] Get "http:https://{out-aws-eks-load_balancer}:port/api/v1/workflows/argo?": EOF

I managed to add argoserver.go support for MaxGRPCMessageSize via ENV variable to argo-server deployment, if exist, if not, use the default of 100MB.
I will have to drop the constant definition and use a func instead to support that.

Our test from above worked with no issues when adding this functionality, with both UI and CLI.

If you think this is the right/good direction, I can open a PR or share the diff, as long as our use case can be supported.
We for sure need to support big workflows, that might scale up to 100's of instances (sometimes even more), 1000's of pods and if we can avoid building argo from source, it will be great.

Many thanks!
David

[alexec]

“argo list” is likely hydrating workflows (ie going to the database to get the node status) when they do not need to. Like the UI, is should pass the “fields” parameter to prevent this.

This should be a one line change here:

argo-workflows/cmd/argo/commands/list.go

Line 112 in 4321259

wfList, err := serviceClient.ListWorkflows(ctx, &workflowpkg.WorkflowListRequest{Namespace: flags.namespace, ListOptions: listOpts})

Should be like this:

argo-workflows/ui/src/app/shared/services/workflows-service.ts

Line 26 in 4321259

fields = [

[alexec]

It’d be great if you want to submit and PR and I expect we can then backport to v3.1.2

[dpeer6]

Cool, then!
I will work on it tomorrow, hope you will find it suitable.

[dpeer6]

Hi,
I created the PR, hope it's ok.

[dpeer6]

I updated the PR using the init func.