-
I have an S3 bucket with 3+ million files in it. I want to build a channel that parses each of the files (understanding that this can take a long time to run) but I cannot get through even polling every message in the bucket without the channel crashing my server due to memory maxing out. Is it possible to not poll every message or is there a better design for this task that anyone has ideas for? |
Beta Was this translation helpful? Give feedback.
Answered by
pacmano1
May 2, 2024
Replies: 1 comment 1 reply
-
You will have to use a tool that supports pagination. e.g. use the java SDK to call "list objects". You will likely need to use the SDK to get the objects as well. |
Beta Was this translation helpful? Give feedback.
1 reply
Answer selected by
pacmano1
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
You will have to use a tool that supports pagination. e.g. use the java SDK to call "list objects". You will likely need to use the SDK to get the objects as well.