[BEAM-8379] Cache Eviction #10062
[BEAM-8379] Cache Eviction #10062
Conversation
KevinGG
force-pushed
the
BEAM-8379
branch
2 times, most recently
from
af08a2b
to
5bb263e
Compare
|
Run Python PreCommit |
KevinGG
force-pushed
the
BEAM-8379
branch
3 times, most recently
from
87994bd
to
5127c87
Compare
1. Implemented cache eviction whenever Python interpreter exits. 2. Cache for PCollections is grouped by PCollections as the Interactive Beam user flow is now data-centric. And cache including its eviction is managed by a global interactive environment instance created/retrieved/reset implicitly by runners in the same main thread/loop. 3. Unified Python version requirement to 3.6 for interactive features and their tests for simplicity. We don't have test suite between py35 and py36 (both exclusive) anyway, so there is no need to check by 3.5.3 when some mock test features are only available in 3.6 and later.
|
R: @pabloem |
|
R: @davidyan74 Thanks! |
| self._is_py_version_ready = False | ||
| logging.warning('Interactive Beam requires Python 3.5.3+.') | ||
| logging.warning('Interactive Beam requires Python 3.6+.') |
There was a problem hiding this comment.
Out of curiosity, why does it require 3.6+?
There was a problem hiding this comment.
The assertions in unittest module for if something is called such as assert_called are not available in some older Python versions.
To make it even worse, some features introduced in Python3 has been patched back to some Python2 versions but not some older Python3 versions. So it's possible to see tests passing in Py3.6+ and Py2.7+ but fail for Py3.5.
The 3.5.3 is only a prerequisite for dependency jsons (not json) that allows us to serialize any Python object into json format without providing schema of the data. That's how we can build dataframe from arbitrary data implicitly.
There was a problem hiding this comment.
So it sounds like the interactive functionality is available in 3.5.3, but the particular assert method we use in unit tests is only available on 3.6+, right? Would it make sense for us to support 3.5.3+, and skip those tests on <3.6? If we force 3.6+ for support, we'd lose 3.5.3+ users, right?
Perhaps it's fine to do only 3.6+, but just wanting to double click on this.
| self._is_py_version_ready = False | ||
| logging.warning('Interactive Beam requires Python 3.5.3+.') | ||
| logging.warning('Interactive Beam requires Python 3.6+.') |
There was a problem hiding this comment.
So it sounds like the interactive functionality is available in 3.5.3, but the particular assert method we use in unit tests is only available on 3.6+, right? Would it make sense for us to support 3.5.3+, and skip those tests on <3.6? If we force 3.6+ for support, we'd lose 3.5.3+ users, right?
Perhaps it's fine to do only 3.6+, but just wanting to double click on this.
Yes! Since we don't have tests for 3.5.3 but only 3.5, 3.6, 3.7, so we'll just say 3.6 in the warning and check for 3.6 in tests for simplicity. |
Interactive Beam user flow is now data-centric. And cache including
its eviction is managed by a global interactive environment instance
created/retrieved/reset implicitly by runners in the same main
thread/loop.
and their tests for simplicity. We don't have test suite between py35
and py36 (both exclusive) anyway, so there is no need to check by
3.5.3 when some mock test features are only available in 3.6 and later.
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
R: @username).[BEAM-XXX] Fixes bug in ApproximateQuantiles, where you replaceBEAM-XXXwith the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.See the Contributor Guide for more tips on how to make review process smoother.
Post-Commit Tests Status (on master branch)
Pre-Commit Tests Status (on master branch)
See .test-infra/jenkins/README for trigger phrase, status and link of all Jenkins jobs.