feature/tree-sources-staging #24
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Added a new merge step to the ETL process. The new merge step takes all the data from the geojson file and moves it into a staging table called `tree_sources_staging`. This is a way to keep the raw data from the sources into a database table. This PR uses ogr2ogr to move data into the database. By default, data is moved 20k rows at a time. Postgres is all lowercase, and javascript uses camelcase, so we cannot easily use underscores in names without some bigger changes. However, that did not seem to be an issue. feature/tree-sources-staging Removed unused new library
zoobot
reviewed
Feb 21, 2023
zoobot
reviewed
Feb 21, 2023
zoobot
approved these changes
Feb 21, 2023
- added a new PG_USE_COPY argument. this improves performance by about 2-3x. - cleaned up command to have shorter strings - moved PG config to a separate variable
|
Feel free to merge this when you are ready @tzinckgraf . I think you have to resolve all conversations. Then use squash and merge. Thanks for this awesome contribution!! |
|
@tzinckgraf Is it ready to go in or still in draft? |
|
@zoobot I have to go through and resolve the conversations, then this is good to go. I can have that done by Wednesday night's meeting. Working on getting the demo / PR ready for this Wednesday first. |
zoobot
reviewed
Mar 26, 2023
- update the ogr2ogr command to transform the sql converting camelCase to snake_case - convert database table to treedata_staging - use spawn instead of exec - truncate the staging table as the first step Added a new line for a little clean up
tzinckgraf
commented
Mar 27, 2023
zoobot
reviewed
Mar 29, 2023
- updated idName to idSourceName as per the rest of the code
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Added a new merge step to the ETL process. The new merge step takes all the data from the geojson file and moves it into a staging table called
tree_sources_staging. This is a way to keep the raw data from the sources into a database table.This PR uses ogr2ogr to move data into the database. By default, data is moved 20k rows at a time. Postgres is all lowercase, and javascript uses camelcase, so we cannot easily use underscores in names without some bigger changes. However, that did not seem to be an issue.
feature/tree-sources-staging
Removed unused new library