Hudi connector #7
Conversation
| projection.get().stream() | ||
| .map(HiveColumnHandle.class::cast) | ||
| .collect(toUnmodifiableList())) | ||
| projection.get().stream() |
There was a problem hiding this comment.
revert this and get rid of the whitespace changes?
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.hudi</groupId> | ||
| <artifactId>hudi-hive-sync</artifactId> |
There was a problem hiding this comment.
We need hudi-hive-sync? for querying?
| import io.trino.spi.connector.ConnectorFactory; | ||
|
|
||
| public class HudiPlugin | ||
| implements Plugin |
There was a problem hiding this comment.
this wrapping. bit odd to me. but seems like the convention.
| private HudiPageSourceBuilderFactory() {} | ||
|
|
||
| public static HudiPageSourceBuilder get( | ||
| HoodieFileFormat hudiFileFormat, HudiConfig hudiConfig, HdfsEnvironment hdfsEnvironment, |
There was a problem hiding this comment.
rename hudiBaseFileFormat and lets use this terminology consistently everywhere? including HudiConfig
| hudiConfig, hdfsEnvironment, stats, timeZone, context); | ||
| default: | ||
| throw new HoodieIOException( | ||
| String.format("File format %s is not supported yet", hudiFileFormat)); |
| import static io.trino.plugin.hudi.HudiSessionProperties.getStandardSplitWeightSize; | ||
| import static io.trino.plugin.hudi.HudiSessionProperties.isSizeBasedSplitWeightsEnabled; | ||
|
|
||
| public class HudiSplitBackgroundLoader |
There was a problem hiding this comment.
how much of this class is code reused from HiveSplitBackgroundLoader
| </dependency> | ||
| <dependency> | ||
| <groupId>org.apache.hudi</groupId> | ||
| <artifactId>hudi-common</artifactId> |
There was a problem hiding this comment.
can we do the same trimming around hbase/parquet-avro - we did for Presto, here as well?
| private final ScheduledExecutorService splitLoaderExecutorService; | ||
| private final ScheduledFuture splitLoaderFuture; | ||
|
|
||
| public HudiSplitSource( |
There was a problem hiding this comment.
whats the life cycle of this object? Once per query?
| import static com.google.common.base.Preconditions.checkArgument; | ||
| import static java.util.Objects.requireNonNull; | ||
|
|
||
| public class HudiSplit |
There was a problem hiding this comment.
Modelled after an existing HiveSplit?
| ConnectorSplit connectorSplit, | ||
| ConnectorTableHandle connectorTable, | ||
| List<ColumnHandle> columns, | ||
| DynamicFilter dynamicFilter) |
There was a problem hiding this comment.
Can you point me to where the file pruning happens i.e filtering out files that don't match the filter ?
codope
force-pushed
the
hudi-plugin-cleanup
branch
from
e86685b
to
c988546
Compare
Rebase and resolve conflicts Use cached thread pool for split generation
codope
force-pushed
the
hudi-plugin-cleanup
branch
from
c988546
to
e440e49
Compare
Class view in RFC.