fix(server): serve artifacts directly from disk to support large artifacts #4589
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
bf839f7
to
84f3fbc
Compare
|
This is marked as draft pending the merge of #4579 since it touches the same code. I'll rebase this PR on top of that once it's been merged and make sure tests pass. |
cde7ef8
to
f3adc2b
Compare
alexec
approved these changes
Nov 23, 2020
|
Please mark as "Ready to review' when ready. |
…facts When serving very large artifacts, first loading them into memory can potentially cause the pod to go OOM/crash depending on how much memory is available and what limits have been set. Rather than loading it into memory, we can serve files directly from disk. Fixes argoproj#4588 Signed-off-by: Daniel Herman <[email protected]>
f3adc2b
to
c774944
Compare
alexcapras
pushed a commit
to alexcapras/argo
that referenced
this pull request
Dec 2, 2020
Signed-off-by: [email protected] <[email protected]> feat(ui): Add Template/Cron workflow filter to workflow page. Closes argoproj#4532 (argoproj#4543) Signed-off-by: Tianchu Zhao <[email protected]> feat(executor): Auto create s3 bucket if not present. Signed-off-by: Alex Capras <[email protected]> Apply codegen Signed-off-by: Alex Capras <[email protected]> Add argo-e2e label to test wf Signed-off-by: Alex Capras <[email protected]> chore: Updated stress test YAML (argoproj#4569) Signed-off-by: Alex Collins <[email protected]> docs: Updated kubectl apply command in manifests README (argoproj#4577) Signed-off-by: Stefan Gloutnikov <[email protected]> feat(controller): Make MAX_OPERATION_TIME configurable. Close argoproj#4239 (argoproj#4562) Signed-off-by: Alex Collins <[email protected]> docs: Fix a typo in example (argoproj#4590) Signed-off-by: Takayoshi Nishida <[email protected]> feat(controller): Retry transient offload errors. Resolves argoproj#4464 (argoproj#4482) Signed-off-by: Alex Collins <[email protected]> fix(server): use the correct name when downloading artifacts (argoproj#4579) Signed-off-by: Daniel Herman <[email protected]> fix(server): serve artifacts directly from disk to support large artifacts (argoproj#4589) Signed-off-by: Daniel Herman <[email protected]> fix(executor): Handle sidecar killing in a process-namespace-shared pod (argoproj#4575) Signed-off-by: Daisuke Taniwaki <[email protected]> docs: Add JSON schema for IDE validation (argoproj#4581) Signed-off-by: Paul Brabban <[email protected]> refactor: Use polling model for workflow phase metric (argoproj#4557) Signed-off-by: Simon Behar <[email protected]> Addressing reviewers comments Signed-off-by: Alex Capras <[email protected]> Addressing reviewers comments docs: Minor typo fix (argoproj#4610) Signed-off-by: Paavo Pokkinen <[email protected]> fix(controller): Prevent tasks with names starting with digit to use either 'depends' or 'dependencies' (argoproj#4598) Signed-off-by: terrytangyuan <[email protected]> fix(docs): Bring minio chart instructions up to date (argoproj#4586) Signed-off-by: Ranga Krishnan <[email protected]> fix(executor): Fixed waitMainContainerStart returning prematurely. Closes argoproj#4599 (argoproj#4601) Signed-off-by: fsiegmund <[email protected]> feat(controller): Enhanced artifact repository ref. See argoproj#3184 (argoproj#4458) Signed-off-by: Alex Collins <[email protected]> fix: Null check pagination variable (argoproj#4617) Signed-off-by: Simon Behar <[email protected]> fix: Perform fields filtering server side (argoproj#4595) Signed-off-by: Simon Behar <[email protected]> fix(server): Correct webhook event payload marshalling. Fixes argoproj#4572 (argoproj#4594) Signed-off-by: Alex Collins <[email protected]> feat(ui): Add columns--narrower-height to AttributeRow (argoproj#4371) fix: Fix TestCleanFieldsExclude (argoproj#4625) Signed-off-by: Simon Behar <[email protected]> fix(argo-server): fix global variable validation error with reversed dag.tasks (argoproj#4369) Signed-off-by: chenyu.zheng <[email protected]> fix: derive jsonschema and fix up issues, validate examples dir… (argoproj#4611) Signed-off-by: Paul Brabban <[email protected]> fix(ui): Reference secrets in EnvVars. Fixes argoproj#3973 (argoproj#4419) Signed-off-by: Alejandro Tejera <[email protected]> fix(ui): Fix Snyk issues (argoproj#4631) Signed-off-by: Alex Collins <[email protected]> feat(executor): More informative log when executors do not support output param from base image layer (argoproj#4620) Signed-off-by: terrytangyuan <[email protected]> Codegen patch. Signed off by [email protected] Codegen patch. Signed off by [email protected] Delete test.patch
alexcapras
pushed a commit
to alexcapras/argo
that referenced
this pull request
Dec 2, 2020
Signed-off-by: [email protected] <[email protected]> feat(ui): Add Template/Cron workflow filter to workflow page. Closes argoproj#4532 (argoproj#4543) Signed-off-by: Tianchu Zhao <[email protected]> feat(executor): Auto create s3 bucket if not present. Signed-off-by: Alex Capras <[email protected]> Apply codegen Signed-off-by: Alex Capras <[email protected]> Add argo-e2e label to test wf Signed-off-by: Alex Capras <[email protected]> chore: Updated stress test YAML (argoproj#4569) Signed-off-by: Alex Collins <[email protected]> docs: Updated kubectl apply command in manifests README (argoproj#4577) Signed-off-by: Stefan Gloutnikov <[email protected]> feat(controller): Make MAX_OPERATION_TIME configurable. Close argoproj#4239 (argoproj#4562) Signed-off-by: Alex Collins <[email protected]> docs: Fix a typo in example (argoproj#4590) Signed-off-by: Takayoshi Nishida <[email protected]> feat(controller): Retry transient offload errors. Resolves argoproj#4464 (argoproj#4482) Signed-off-by: Alex Collins <[email protected]> fix(server): use the correct name when downloading artifacts (argoproj#4579) Signed-off-by: Daniel Herman <[email protected]> fix(server): serve artifacts directly from disk to support large artifacts (argoproj#4589) Signed-off-by: Daniel Herman <[email protected]> fix(executor): Handle sidecar killing in a process-namespace-shared pod (argoproj#4575) Signed-off-by: Daisuke Taniwaki <[email protected]> docs: Add JSON schema for IDE validation (argoproj#4581) Signed-off-by: Paul Brabban <[email protected]> refactor: Use polling model for workflow phase metric (argoproj#4557) Signed-off-by: Simon Behar <[email protected]> Addressing reviewers comments Signed-off-by: Alex Capras <[email protected]> Addressing reviewers comments docs: Minor typo fix (argoproj#4610) Signed-off-by: Paavo Pokkinen <[email protected]> fix(controller): Prevent tasks with names starting with digit to use either 'depends' or 'dependencies' (argoproj#4598) Signed-off-by: terrytangyuan <[email protected]> fix(docs): Bring minio chart instructions up to date (argoproj#4586) Signed-off-by: Ranga Krishnan <[email protected]> fix(executor): Fixed waitMainContainerStart returning prematurely. Closes argoproj#4599 (argoproj#4601) Signed-off-by: fsiegmund <[email protected]> feat(controller): Enhanced artifact repository ref. See argoproj#3184 (argoproj#4458) Signed-off-by: Alex Collins <[email protected]> fix: Null check pagination variable (argoproj#4617) Signed-off-by: Simon Behar <[email protected]> fix: Perform fields filtering server side (argoproj#4595) Signed-off-by: Simon Behar <[email protected]> fix(server): Correct webhook event payload marshalling. Fixes argoproj#4572 (argoproj#4594) Signed-off-by: Alex Collins <[email protected]> feat(ui): Add columns--narrower-height to AttributeRow (argoproj#4371) fix: Fix TestCleanFieldsExclude (argoproj#4625) Signed-off-by: Simon Behar <[email protected]> fix(argo-server): fix global variable validation error with reversed dag.tasks (argoproj#4369) Signed-off-by: chenyu.zheng <[email protected]> fix: derive jsonschema and fix up issues, validate examples dir… (argoproj#4611) Signed-off-by: Paul Brabban <[email protected]> fix(ui): Reference secrets in EnvVars. Fixes argoproj#3973 (argoproj#4419) Signed-off-by: Alejandro Tejera <[email protected]> fix(ui): Fix Snyk issues (argoproj#4631) Signed-off-by: Alex Collins <[email protected]> feat(executor): More informative log when executors do not support output param from base image layer (argoproj#4620) Signed-off-by: terrytangyuan <[email protected]> Codegen patch. Signed off by [email protected] Codegen patch. Signed off by [email protected] Delete test.patch Signed-off-by: Alex Capras <[email protected]>
alexec
pushed a commit
that referenced
this pull request
Dec 3, 2020
…facts (#4589) Signed-off-by: Daniel Herman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
When serving very large artifacts, first loading them into memory can potentially
cause the pod to go OOM/crash depending on how much memory is available and what
limits have been set. Rather than loading it into memory, we can serve files
directly from disk.
Fixes #4588