Add support for running the reindex job across resources in multiple partitions and all partitions #6009
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
#6008
What was done:
IJobPartitionProvider
which can return a list of partitions for operation requests. The implementation will handle requests configured withRequestPartitionId.allPartitions
and return a list of all partitions, including the default partition. Please mind that this interface is likely to change as another factor for determining partitions to run a job against can also be a list of urls provided as parameters in these jobs. Further simplification can be made and I plan to add another PR just for that refactoring simplification (bringUrlPartitioner
into the mix as it also computes partitions for jobs).ReindexProvider
to use the new bean to compute the partitionsIModelJson
) were merged and made some of the templating across job steps simpler. I removed the "partitioned" prefix out of the classes since there was no use-case not requiring partitions and it seemed right. For example,RequestDetails
did not get extended when partitioning was added.GenerateRangeChunksStep
andJobPartitions
. Also made updates such that the chunk partitionId is the authoritative source for partition information to handle the next steps for the chunk.GenerateRangeChunksStep
such that can have cartesian product (urls x partitions) for generating chunks provided in the job parameters.