Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement the Buf to avoid extra memory allocation #4065

Closed
WenyXu opened this issue May 29, 2024 · 6 comments · Fixed by #4585
Closed

Implement the Buf to avoid extra memory allocation #4065

WenyXu opened this issue May 29, 2024 · 6 comments · Fixed by #4585
Assignees
@ozewr
Labels
good first issue Good for newcomers

Comments

@WenyXu
Copy link
Member

WenyXu commented May 29, 2024 •
edited
Loading

We need something like this: https://github.com/apache/opendal/blob/dd70b3c706f1689c2fe9455405cbd1e773c483ab/core/src/types/buffer.rs#L114-L122

Originally posted by @WenyXu in #4038 (comment)
@WenyXu WenyXu added the good first issue Good for newcomers label May 29, 2024
@LYZJU2019
Copy link
Contributor

Can I give it a try?

@WenyXu
Copy link
Member Author

WenyXu commented May 29, 2024

Can I give it a try?

Have fun

LYZJU2019 added a commit to LYZJU2019/greptimedb that referenced this issue Jun 2, 2024
@LYZJU2019
Implement the Buf to avoid extra memory allocation GreptimeTeam#4065
3c0b73e
@LYZJU2019 LYZJU2019 mentioned this issue Jun 2, 2024
Closed
3 tasks
@ozewr
Copy link
Contributor

ozewr commented Aug 17, 2024

Can I give it a try?

@WenyXu
Copy link
Member Author

WenyXu commented Aug 19, 2024

Can I give it a try?

Have fun🚀

@ozewr
Copy link
Contributor

ozewr commented Aug 19, 2024

Sorry,this question unrelated to this issue logically.What is the minimum memory required to compile this project? I encountered an out-of-memory (OOM) issue when running cargo nextest run. I was running it inside a Docker container with the configuration: --cpus=8 --memory 16g --memory-swap=32g. I also tried running it directly on my personal PC, but it caused the system to crash. My PC has 24GB of RAM.I can't run it 😢.@WenyXu

[12662.902005] Memory cgroup out of memory: Killed process 211599 (ld) total-vm:2666868kB, anon-rss:2659840kB, file-rss:3780kB, shmem-rss:0kB, UID:0 pgtables:5276kB oom_score_adj:0

@WenyXu
Copy link
Member Author

WenyXu commented Aug 19, 2024

Sorry,this question unrelated to this issue logically.What is the minimum memory required to compile this project? I encountered an out-of-memory (OOM) issue when running cargo nextest run. I was running it inside a Docker container with the configuration: --cpus=8 --memory 16g --memory-swap=32g. I also tried running it directly on my personal PC, but it caused the system to crash. My PC has 24GB of RAM.I can't run it 😢.@WenyXu

[12662.902005] Memory cgroup out of memory: Killed process 211599 (ld) total-vm:2666868kB, anon-rss:2659840kB, file-rss:3780kB, shmem-rss:0kB, UID:0 pgtables:5276kB oom_score_adj:0

Hi, you can try this https://github.com/rui314/mold

@ozewr ozewr mentioned this issue Aug 19, 2024
Merged
3 tasks
ozewr pushed a commit to ozewr/greptimedb that referenced this issue Aug 19, 2024
remove todo GreptimeTeam#4065
5524980
github-merge-queue bot pushed a commit that referenced this issue Aug 19, 2024
@ozewr @WenyXu
feat: Implement the Buf to avoid extra memory allocation (#4585)
30af787 
* feat: Implement the Buf to avoid extra memory allocation

* fmt toml

* fmt code

* mv entry.into_buffer to raw_entry_buffer

* less reuse opendal

* remove todo #4065

* Update src/mito2/src/wal/entry_reader.rs

Co-authored-by: Weny Xu <[email protected]>

* fmt code

---------

Co-authored-by: ozewr <[email protected]>
Co-authored-by: Weny Xu <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ozewr ozewr

Labels
good first issue Good for newcomers
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

feat: Implement the Buf to avoid extra memory allocation ozewr/greptimedb
3 participants
@WenyXu @LYZJU2019 @ozewr