[Data] Add override_num_blocks parameter to get_parquet_metadata_provider
#45767
Conversation
Signed-off-by: Balaji Veeramani <[email protected]>
| @@ -13,7 +13,7 @@ def get_generic_metadata_provider(file_extensions: Optional[List[str]]): | |||
| return DefaultFileMetadataProvider() | |||
|
|
|||
|
|
|||
| def get_parquet_metadata_provider(): | |||
There was a problem hiding this comment.
actually, would it be better to add all read_parquet parameters?
There was a problem hiding this comment.
or pass in the ParquetDatasource as the parameter?
There was a problem hiding this comment.
actually, would it be better to add all read_parquet parameters?
What would the motivation for this? To make the signature less arbitrary?
or pass in the ParquetDatasource as the parameter?
ParquetDatasource doesn't have the value of override_num_blocks, so I'm not sure if that'd work.
There was a problem hiding this comment.
What would the motivation for this? To make the signature less arbitrary?
just in case we'll need more info for determining metadata provider in the future.
Not a big deal though, if it requires too much work, we can revisit this later.
There was a problem hiding this comment.
Gotcha. Given that we might not even use the _default_metadata_provider approach going forward, let's revisit later
…rovider` (ray-project#45767) Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]>
…rovider` (ray-project#45767) Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: yucai <[email protected]>
…rovider` (ray-project#45767) Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Ryan O'Leary <[email protected]>
…rovider` (ray-project#45767) Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: yucai <[email protected]>
…rovider` (ray-project#45767) Signed-off-by: Balaji Veeramani <[email protected]> Signed-off-by: Richard Liu <[email protected]>
Why are these changes needed?
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.