Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Delete series for object storage. #1598

Closed
bwplotka opened this issue Oct 2, 2019 · 55 comments
Closed

Delete series for object storage. #1598

bwplotka opened this issue Oct 2, 2019 · 55 comments
Labels
difficulty: hard feature request/improvement GSoC/Community Bridge/LFX priority: P0 stale

Comments

@bwplotka
Copy link
Member

bwplotka commented Oct 2, 2019

Goals:

  • Allow Delete series from blocks kept in an object storage

Use cases:

Related:

It's a dry issue, design to come. (:
@bwplotka
Copy link
Member Author

bwplotka commented Oct 3, 2019 •
edited
Loading

Challenges:

  • How to handle long-running job for that? Should it be compactor? How to coordinate with the compactor if not? How to measure progress?
  • Is it even worth to add tombstones logic?
    • For GDPR we care of data complete removal, not just unvisible
    • For aggr retention as well - we care about data size. Tombstones will only slow all down and add size.

cc @krasi-georgiev

@squat squat mentioned this issue Jan 8, 2020
Closed
@SuperQ
Copy link
Contributor

SuperQ commented Jan 21, 2020

My opinion:

  • The compactor seems like the right place for this.
  • I think following the tombstones logic is fine. It works well for Prometheus itself.
  • Adding the clean_tombstones endpoint/function is probably the right way to deal with purging if necessary.

@bwplotka
Copy link
Member Author

Yup @SuperQ

As we talked offline tombstones are probably no go, as they are not immutable. At least not in the form that is implemented in Prometheus. We might be though able to add deletion requests to each block, then having a compactor to include that during compaction and lazy rewrite. It would be nice to have some solution that will also fit #903

@bwplotka bwplotka mentioned this issue Jan 29, 2020
Open
@stale
Copy link

stale bot commented Feb 24, 2020

This issue/PR has been automatically marked as stale because it has not had recent activity. Please comment on status otherwise the issue will be closed in a week. Thank you for your contributions.

@stale stale bot added the stale label Feb 24, 2020
@bwplotka
Copy link
Member Author

Still valid. Help wanted.

@stale stale bot removed the stale label Feb 24, 2020
@GiedriusS GiedriusS mentioned this issue Mar 4, 2020
Closed
@SidharthBansal
Copy link

Hello, I am looking forward to contributing to this for GSoC 2020. How shall I start?

@infa-kparida
Copy link

This will be a super useful feature for decommissioning prometheus clusters.Looking forward to it +1

@bwplotka
Copy link
Member Author

bwplotka commented Apr 8, 2020 •
edited
Loading

Good point!

Looks like we did not find anyone to help us on this with GSoC, we will try with Summer Community Bridge. (:

So help wanted (:

@SidharthBansal
Copy link

What is the timeline for Summer Community Bridge and what is the link where I can apply?

@bwplotka
Copy link
Member Author

bwplotka commented Apr 8, 2020

@idvoretskyi might know more.

@idvoretskyi
Copy link

@bwplotka we'll announce this later in April/early May.

@SidharthBansal
Copy link

SidharthBansal commented Apr 8, 2020 via email

@idvoretskyi
Copy link

@SidharthBansal in the meantime, feel to track the repo https://github.com/cncf/soc/tree/master/communitybridge; we'll post the updates there.

@SidharthBansal
Copy link

SidharthBansal commented Apr 8, 2020 via email

@idvoretskyi
Copy link

@SidharthBansal we don't have any specific college-related restrictions.

Also, please see - https://docs.linuxfoundation.org/docs/communitybridge-mentorship/mentees#Mentees-EligibilityRules.

@SidharthBansal
Copy link

SidharthBansal commented Apr 8, 2020 via email

@SidharthBansal
Copy link

SidharthBansal commented Apr 8, 2020 via email

@idvoretskyi
Copy link

@SidharthBansal I assume this discussion is out of scope for this repo.

Please raise your questions by opening an issue at https://github.com/cncf/soc, or directly at https://support.communitybridge.org/.

@SidharthBansal
Copy link

SidharthBansal commented Apr 8, 2020 via email

@bwplotka bwplotka mentioned this issue Apr 16, 2020
Closed
@stale stale bot added the stale label Jan 4, 2021
@Harshitha1234
Copy link
Contributor

Still valid!

@stale stale bot removed the stale label Jan 5, 2021
@bwplotka
Copy link
Member Author

bwplotka commented Feb 1, 2021 •
edited
Loading

This is already doable by thanos tools rewrite (: Should we close?

@bwplotka
Copy link
Member Author

bwplotka commented Feb 1, 2021

Ideally we can focus on tracking automation and embedding into API if we want as @Harshitha1234 started in separate issues 🤗

@bwplotka bwplotka closed this as completed Feb 1, 2021
@kesor
Copy link

kesor commented Feb 14, 2021

This is already doable by thanos tools rewrite (: Should we close?

How exactly is this doable? Is there a link to the documentation of how to do delete_series using thanos tools rewrite? Have been finding plenty of issues asking for this feature, and now it says its "done", but there is zero information on how to actually do it.

@huyifeng0522
Copy link

This is already doable by thanos tools rewrite (: Should we close?

How exactly is this doable? Is there a link to the documentation of how to do delete_series using thanos tools rewrite? Have been finding plenty of issues asking for this feature, and now it says its "done", but there is zero information on how to actually do it.

Can not agree more，I tried thanos tools rewrite but it did not work

@yeya24
Copy link
Contributor

yeya24 commented Sep 25, 2021

Sorry for the delay. I will add a document to demonstrate how to use the rewrite tool to delete and relabel series.

@yeya24
Copy link
Contributor

yeya24 commented Sep 25, 2021

PTAL for pr #4701.

@yeya24
Copy link
Contributor

yeya24 commented Oct 11, 2021 •
edited
Loading

Though this is something supported already by the bucket rewrite tool, I still think it useful to implement the tombstone approach like #3075. The code is there and someone can take it over. Once the comments and performance issues are addressed, then I think it is good to go.

I feel this can coexist with bucket rewrite. Tombstone file is doing some series filtering online so query performance will be impacted a little bit and the deletion can be applied lazily (compactor), no whole block modification is required.

I am still interested in this so I want to reopen this issue. It is also okay to track this at another issue if it is better and clearer.

@yeya24 yeya24 reopened this Oct 11, 2021
@stale
Copy link

stale bot commented Jan 9, 2022

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

@stale stale bot added the stale label Jan 9, 2022
@GiedriusS GiedriusS removed the stale label Jan 10, 2022
@stale
Copy link

stale bot commented Apr 16, 2022

Hello 👋 Looks like there was no activity on this issue for the last two months.
Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗
If there will be no activity in the next two weeks, this issue will be closed (we can always reopen an issue if we need!). Alternatively, use remind command if you wish to be reminded at some point in future.

@stale stale bot added the stale label Apr 16, 2022
@stale
Copy link

stale bot commented Apr 30, 2022

Closing for now as promised, let us know if you need this to be reopened! 🤗

@stale stale bot closed this as completed Apr 30, 2022
@matej-g matej-g mentioned this issue Sep 21, 2022
Open
6 tasks
@BouchaaraAdil
Copy link
Contributor

still valid!

@Ishmeet
Copy link

Ishmeet commented Jul 10, 2023

is there an api ?

@hartfordfive
Copy link

Any updates on this issue? I would like to know how I can manually delete series easily in the bucket without having to run compactor?

@MichaHoffmann
Copy link
Contributor

Any updates on this issue? I would like to know how I can manually delete series easily in the bucket without having to run compactor?

I dont think we have anything for that yet!

@Kot-o-pes
Copy link

Hello, ive come across running out of s3 space and unfortunately there is no way to delete blocks by label, instead i need to get id of each needed block and then mark it by id, it would be nice lets say to add flag -l label=value to thanos tools mark

@gillg
Copy link

gillg commented Apr 23, 2024

Could be interesting indeed, but the only filter is the tenant as I'm aware.
But side question, what do you mean by "running out of S3 space" ?? It's literally unlimited, what is your limit?

@Kot-o-pes
Copy link

Could be interesting indeed, but the only filter is the tenant as I'm aware. But side question, what do you mean by "running out of S3 space" ?? It's literally unlimited, what is your limit?

Its a self hosted storage (minio), while cleaning the space i thought that it would be nice to have different retention for tenants, for example i have 1 bucket and 3 tenants (prod,stage,dev) and it would make life much easier if i could set up different retentions for different tenants, to achieve storing longer periods for prod and shorter for dev without having to set up three thanos installations with 3 different buckets

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
difficulty: hard feature request/improvement GSoC/Community Bridge/LFX priority: P0 stale
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests