Add new api to upload csv, table.yaml, and query into HDFS #48
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Collaborator
NLGithubWP
commented
May 29, 2022
•
- Enable provide .dz file name in the loading process.
- Add distributed APIs to enable users to load local CSV and upload it to HDFS.
- Add distributed APIs to enable users to upload queries.json files, store them locally, and upload them into HDFS.
- Add comments.
- Add Client Package to send reqeust.
- docs is at Add distributed APIs into doc COOL-site#23
KimballCai
requested changes
Jun 20, 2022
| @@ -9,7 +9,7 @@ | |||
| public class CliDataWriter implements DataWriter { | |||
|
|
|||
| @Override | |||
| public boolean Initialize() throws IOException { | |||
| public boolean Initialize(String... fileName) throws IOException { | |||
There was a problem hiding this comment.
Please use String[] instead of the String...
| @@ -137,8 +137,15 @@ private void finishCublet() throws IOException { | |||
| * @return DataOutputStream | |||
| * @throws IOException | |||
| */ | |||
| private DataOutputStream newCublet() throws IOException { | |||
| String fileName = Long.toHexString(System.currentTimeMillis()) + ".dz"; | |||
| private DataOutputStream newCublet(String... fileNameParams) throws IOException { | |||
There was a problem hiding this comment.
why do we need to give a file name?
| @@ -33,6 +33,10 @@ public class LoadQuery { | |||
| private String outputPath; | |||
| private String configPath; | |||
|
|
|||
| // after writing dz, table file, records the path inside the server. | |||
| private String dzFilePath; | |||
| private String TableFilePath; | |||
There was a problem hiding this comment.
TableFilePath should be the same as the schemaPath.
| @@ -33,6 +33,10 @@ public class LoadQuery { | |||
| private String outputPath; | |||
| private String configPath; | |||
|
|
|||
| // after writing dz, table file, records the path inside the server. | |||
| private String dzFilePath; | |||
There was a problem hiding this comment.
Why do we need the dzFilePath?
| * @param dataSourceName output cube name. Need to be specified when loading from the repository, eg, sogamo | ||
| * @param schemaFileName path to the table.yaml, eg. sogamo/table.yaml | ||
| * @param dataFileName path to the data.csv, eg. sogamo/test.csv | ||
| * @param cubeRepo the name of the output cube repository. eg. datasetSource |
There was a problem hiding this comment.
It needs a @param for fileName
| public static void main(String[] args) throws Exception { | ||
|
|
||
| if (args.length != 1) { | ||
| System.err.println("Pass in query id (Example: q1)"); |
| HDFSConnection fs = HDFSConnection.getInstance(); | ||
|
|
||
| String localPath1 = req.getOutputPath() + "/" + req.getDzFilePath();; | ||
| String dfsPath1 = "/cube/" + req.getDzFilePath(); |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.