-
Notifications
You must be signed in to change notification settings - Fork 6.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Keep track of obsolete blob files in VersionSet #6755
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
… the obsolete list
…was ineffective otherwise) and add one more to make sure a file is only returned once
2b2f11c
to
f0a08ea
Compare
@ltamasi has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM. Thanks @ltamasi for adding this.
@ltamasi has updated the pull request. Re-import the pull request |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@ltamasi has imported this pull request. If you are a Facebook employee, you can view this diff on Phabricator.
Summary: The patch adds logic that returns the set of live blob files from `Version::AddLiveFiles` and `VersionSet::AddLiveFiles` (in addition to live table files), and also cleans up the code a bit, for example, by exposing only the numbers of table files as opposed to the earlier `FileDescriptor`s that no clients used. Moreover, the patch extends the `GetLiveFiles` API so that it also exposes blob files in the current version. Similarly to #6755, this is a building block for identifying and purging obsolete blob files. Pull Request resolved: #6785 Test Plan: `make check` Reviewed By: riversand963 Differential Revision: D21336210 Pulled By: ltamasi fbshipit-source-id: fc1aede8a49eacd03caafbc5f6f9ce43b6270821
Summary:
The patch adds logic to keep track of obsolete blob files. A blob file becomes
obsolete when the last
shared_ptr
that points to the correspondingSharedBlobFileMetaData
object goes away, which, in turn, happens when thelast
Version
that contains the blob file is destroyed. No longer needed blobfiles are added to the obsolete list in
VersionSet
using a custom deleter toavoid unnecessary coupling between
SharedBlobFileMetaData
andVersionSet
.Obsolete blob files are returned by
VersionSet::GetObsoleteFiles
and storedin
JobContext
.Test Plan:
make check