[FLINK-10778] [tests] Extend TypeSerializerSnapshotMigrationTestBase to test migrating from multiple versions #7504
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
That utility class had a single helper method, restoreFromSnapshot, which accepts the target snapshot's Flink version. This was useful before, because the way to restore snapshots was a bit different for Flink <= 1.1 and newer versions. Since we now no longer support compatibility for 1.1 versions and below, this helper method is simply forwarding the restore operation to the test harness. This commit refactors this have equivalent behaviour directly in the AbstractStreamOperatorTestHarness class.
tzulitai
force-pushed
the
FLINK-10778
branch
2 times, most recently
from
41a547f
to
7f9c4f1
Compare
…are of snapshot version Before, the tests in TypeSerializerSnapshotMigrationTestBase always assumed that the test serializer snapshotw were written with Flink 1.6. This commit makes this flexible, so that we can allow subclasses to specify different snapshot versions for each TestSpecification.
…exible in TypeSerializerSnapshotMigrationTestBase Before, the TypeSerializerSnapshotMigrationTestBase test base always only asserted that a new serializer is compatible as is with the previous serializer's snapshot. This makes the test code not resusable for cases where the new serializer provided isn't compatible as is, but other compatibility types, for example Kryo serializers that requires reconfiguration on restore. This allows the test base to express those cases.
…zer test in TypeSerializerSnapshotMigrationTestBase The serializerSnapshotRestoresCurrentSerializer test asserts that the restored serializer instance from the snapshot is of the same type as the current serializer. This isn't necessary true, as is not a contract of the TypeSerializerSnapshot abstraction. Therefore, this test is removed.
…ild test specifications for multiple test versions
…to test restoring from 1.7 This commit also refactors the BaseTypeSerializerSnapshotMigrationTest by using the new TypeSerializations utility class, to reduce the amount of duplicate code.
…nTest to test restoring from 1.7 This commit also refactors the PrimitiveArraySerializerSnapshotMigrationTest by using the new TestSerializations utility class, to reduce the amount of duplicate code.
…Test to test restoring from 1.7
…est restoring from 1.7
…st restoring from 1.7
…to test restoring from 1.7
…o test restoring from 1.7
…est to test restoring from 1.7
… to test restoring from 1.7
…st to test restoring from 1.7
|
Since Travis is green and we've already went through reviews offline with @igalshilman, will proceed to merge this .. |
tisonkun
pushed a commit
to tisonkun/flink
that referenced
this pull request
Jan 17, 2019
…ase subclasses to test restoring from 1.7 This closes apache#7504.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
What is the purpose of the change
Prior to this PR, the
TypeSerializerSnapshotMigrationTestBasewas hardcoded to only test restoring from serializer snapshots generated in Flink 1.6.x.The main goal of this PR is to:
TypeSerializerSnapshotabstraction.Brief change log
MigrationVersionclass to flink-core, since it'll be used byTypeSerializerSnapshotMigrationTestBase.TypeSerializerSnapshotMigrationTestBaseversion-aware. Test specifications can now define the version that the target test snapshot was taken in.TypeSerializerSnapshotabstraction.TestSpecificationsclass that would help with de-duplicating code when specifying test specifications for multiple versions.MigrationVersion.1.7to the parameters of all existingTypeSerializerSnapshotMigrationTestBasesubclasses, as well as generating the required test snapshot / data files for them.Verifying this change
This is a refactoring of tests.
All subclasses of
TypeSerializerSnapshotMigrationTestBaseshould be passing still.Does this pull request potentially affect one of the following parts:
@Public(Evolving): noDocumentation