-
Notifications
You must be signed in to change notification settings - Fork 1.5k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[freshness-refactor][2/n] Reorganize/Rename/Remove/Document CachingInstanceQueryer methods #12813
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎ |
# RECONCILIATION | ||
#################### | ||
|
||
def get_latest_storage_id(self, event_type: DagsterEventType) -> Optional[int]: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is get_latest_storage_id
particularly related to reconciliation?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
reasonable question -- considering this function is not actually caching anything and is only used once (in asset backfill land), it's debatable if it seems like it might be more appropriate to move this out of the class entirely.
6651e74
to
de8df27
Compare
Summary & Motivation
This does a few things:
_no_materializations_after_cursor_cache
from the class. Instead, to answer queries of the form "what is the latest materialization for asset_partition Y after cursor X", we simply look at the latest materialization in general for asset_partition Y, then compare its record_id to the cursor. If the cursor is greater than that record id, we know there are no materializations after that cursor, and return None. If the cursor is less than that record id, the latest materialization after that cursor is the latest overall materialization.cached_method
to be of the form deffoo(self, *, arg1, arg2)
(stole this from @smackesey). This properly indicates to the user/type system that all arguments must be passed in as kwargs.How I Tested These Changes
Unscientific testing seems to indicate that this has either zero or mildly-positive performance impact