Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

2957 append data to table using existing schema #2969

Merged

Conversation

entantoencuanto
Copy link
Member

@entantoencuanto entantoencuanto commented Apr 14, 2020

Closes #2957

✌️ What does this PR do?

  • Fixes an exception generated when write queries are sent to module database and configuration for queries with writing permissions fails.
  • Makes custom schema parsing more flexible to detect custom configuration for CSV columns using a case-insensitive lookup.
  • Makes import from CSV dataset method to make use of a custom schema based on existing table columns types when used with append option enabled and no schema configuration is provided.
  • Changes the default date format importing from csv to "YYY-MM-DD"
  • Allows the omission of original_name in schema JSON file. If omitted it's assumed that the
    original_name the same as the destination name
  • Adds some tests

:shipit: Does this PR changes any configuration file?

No

(Changes in these files might need to update the role in Ansible)

📖 Does this PR require updating the documentation?

No

@ferblape
Copy link
Member

Looking good. Let me know when it's available in staging because I need it for a few datasets 🎸

@entantoencuanto entantoencuanto marked this pull request as ready for review April 15, 2020 14:19
@entantoencuanto
Copy link
Member Author

Ready in staging!

@@ -57,7 +57,7 @@ def complete_schema_with_defaults(source_file, schema_definition)
default_schema = inspect_csv_schema(source_file, csv_separator: @csv_separator)
schema_definition = (schema_definition || {}).deep_symbolize_keys
default_schema.map do |default_column, default_value|
schema_definition.find { |_, value| value[:original_name] == default_value[:original_name] } || [default_column, default_value]
schema_definition.find { |_, value| value[:original_name].casecmp?(default_value[:original_name]) } || [default_column, default_value]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why the original_name is mandatory? I think it should be optional if the name doesn't change

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changed, now this attribute can be omitted

#
def test_dataset_update_with_file_upload_append_with_schema
with(site: site) do
with_stubbed_jwt_default_secret(default_secret) do

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Block has too many lines. [29/25]

# PUT /api/v1/data/datasets/dataset-slug
#
def test_dataset_update_with_file_upload_append_with_schema
with(site: site) do

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Block has too many lines. [31/25]

# PUT /api/v1/data/datasets/dataset-slug
#
def test_dataset_update_with_file_upload_append
with(site: site) do

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Block has too many lines. [27/25]

# PUT /api/v1/data/datasets/dataset-slug
#
def test_dataset_update_with_file_upload
with(site: site) do

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Block has too many lines. [27/25]

@entantoencuanto entantoencuanto merged commit af49c83 into master Apr 16, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Data / Appending data to an existing dataset with custom schema fails
3 participants