Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Decoupling local cache function and cache algorithm #38048

Merged

Conversation

KinderRiven
Copy link
Contributor

@KinderRiven KinderRiven commented Jun 14, 2022

Changelog category (leave one):

  • Not for changelog (changelog entry is not required)

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

  • Resolve previous issues at PR
  • Decoupling the caching algorithm and caching function, we implement the caching algorithm based on IFilecachePriority, thus avoiding the modification of FileCache after implementing a new algorithm.
  • Currently only LRU algorithm is supported, maybe more cache algorithm in future.

Information about CI checks: https://clickhouse.com/docs/en/development/continuous-integration/

@robot-clickhouse robot-clickhouse added the pr-not-for-changelog This PR should not be mentioned in the changelog label Jun 14, 2022
@KinderRiven KinderRiven changed the title Decoupling cache function and algorithm Decoupling local cache function and cache algorithm Jun 14, 2022
@qoega qoega added the can be tested Allows running workflows for external contributors label Jun 14, 2022
@kssenii kssenii self-assigned this Jun 15, 2022
src/Common/LRUFileCache.h Outdated Show resolved Hide resolved
src/Common/LRUFileCache.h Outdated Show resolved Hide resolved
@KinderRiven KinderRiven force-pushed the decoupling_cache_function_and_algorithm branch from 788aa1c to 0c76d56 Compare June 25, 2022 17:28
@KinderRiven KinderRiven requested a review from kssenii June 26, 2022 15:58
src/Common/FileCache.h Outdated Show resolved Hide resolved
src/Common/LRUFileCache.h Outdated Show resolved Hide resolved
src/Common/FileCache.h Outdated Show resolved Hide resolved
@KinderRiven KinderRiven requested a review from kssenii June 28, 2022 09:02
src/Common/LRUFileCachePriority.h Outdated Show resolved Hide resolved
src/Common/LRUFileCachePriority.h Outdated Show resolved Hide resolved
src/Common/LRUFileCachePriority.h Outdated Show resolved Hide resolved
src/Common/IFileCachePriority.h Outdated Show resolved Hide resolved
src/Common/FileCacheType.h Outdated Show resolved Hide resolved
src/Common/FileCache.cpp Outdated Show resolved Hide resolved
src/Common/FileCache.cpp Outdated Show resolved Hide resolved
src/Common/LRUFileCachePriority.h Outdated Show resolved Hide resolved
src/Common/LRUFileCachePriority.h Outdated Show resolved Hide resolved
src/Common/FileCache.cpp Show resolved Hide resolved
@KinderRiven KinderRiven requested a review from kssenii June 30, 2022 04:20
src/Common/LRUFileCachePriority.h Show resolved Hide resolved
src/Common/LRUFileCachePriority.h Outdated Show resolved Hide resolved
src/Common/FileCache.cpp Outdated Show resolved Hide resolved
src/Common/FileCache.cpp Outdated Show resolved Hide resolved
src/Common/FileCache.cpp Outdated Show resolved Hide resolved
src/Common/FileCache.cpp Show resolved Hide resolved
src/Common/FileCache.h Show resolved Hide resolved
@KinderRiven KinderRiven force-pushed the decoupling_cache_function_and_algorithm branch from 26cab54 to 59489e7 Compare July 28, 2022 05:24
@kssenii kssenii mentioned this pull request Aug 4, 2022
@KinderRiven KinderRiven force-pushed the decoupling_cache_function_and_algorithm branch 3 times, most recently from 6d23d8c to b498a1b Compare August 10, 2022 05:52
class IIterator;
using Key = FileCacheKey;
using ReadIterator = std::shared_ptr<const IIterator>;
using WriteIterator = std::shared_ptr<IIterator>;
Copy link
Member

@kssenii kssenii Aug 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be unique_ptr? or it is shared somewhere?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ReadIterator can be unique_ptr. But for WriteIterator, I don't know if it should be set to unique_ptr, because it may be copied with the FileSegment, do we need to invalidate the iterator before copying?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

because it may be copied with the FileSegment, do we need to invalidate the iterator before copying?

but file segment cannot be copied

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this, I tried to modify it, but sometimes we need to use a temporary pointer to keep some code readable, such as the following cases, so I think it may be better to keep it as it is, what do you think?

auto priority_iter = record->second;
priority_iter->use(cache_lock);
result_state = priority_iter->hits() >= enable_cache_hits_threshold ? FileSegment::State::EMPTY : FileSegment::State::SKIP_CACHE;

Copy link
Member

@kssenii kssenii Aug 10, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

but looks like here auto & is possible, then unique_ptr will be ok

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried to fix it, but I always get an error when compiling involves unordered_map. Maybe there is something wrong with my understanding of uniqueu_ptr :(

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

may be not use a pointer at all - just an object? (and in FileSegmentCell put std::optional to iterator).

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sounds feasible, do you mean all ReadIterator and WriteIterator as objects? In this case, there will be more revisions, but I may not have much time to make revisions recently (there is a lot of work to deal with in reality), and I will reconsider the relevant implementation when I have time :).

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do you mean all ReadIterator and WriteIterator as objects

yes

but I may not have much time to make revisions recently (there is a lot of work to deal with in reality), and I will reconsider the relevant implementation when I have time

ok, let's leave it as it is for now

Copy link
Member

@kssenii kssenii left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM
thanks, very good changes

@KinderRiven KinderRiven force-pushed the decoupling_cache_function_and_algorithm branch from 1a23a95 to 1aa7bbc Compare August 10, 2022 16:12
@kssenii kssenii merged commit be69169 into ClickHouse:master Aug 11, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
can be tested Allows running workflows for external contributors pr-not-for-changelog This PR should not be mentioned in the changelog
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants