Census Repository data model and CSV import behavior #69
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Connects to #55.
What does this PR do?
This PR implements a starting data model and back-end for the coming Census-related features. The first of them consists on the ingestion of data from CSV files, that is already implemented as part of this PR.
Regarding that Census data model and back-end, there're a few concepts that come into play. Let me introduce them:
CensusItem
: Represents a Census record in the system. Note that it doesn't have any constraints, validations or integrity rules because it is intended to just behave like a system-agnostic storage.Admin::CensusImport
: Represents any Census Importing processes through the Admin namespace. A new record from this class is created every time an Admin user performs a CSV (for now) importing process, and stores its output to be used in coming imports. It is used as an archaic versioning system (everyCensusItem
belongs to aAdmin::CensusImport
process), as well as to summarize some import details in the proper view.CensusRepository
: This is an abstraction to provide a common interface to any Census related operations. It is taking care of adapting parameters and attributes to, in example, deal with the hasheddocument_number
field atCensusItem
level. It is also providing validations instead of doing so at AR Model level so the actual back-end or storage behind remains transparent to us.SecretAttribute
: That's a quite simple library to apply a hashing algorithm against any of our model attributes. Not much else to say.Admin::CensusImportForm
: This one keeps all the logic behind the CSV importing process, including parsing, item building (viaCensusRepository
), CensusImport tracking (viaAdmin::CensusImport
) and deletion of old records (from previousAdmin::CensusImport
processes).How should this be manually tested?
The importing form can be found at https://gobierto.dev/admin/census/imports/new, that is also reachable form the Users management view (https://gobierto.dev/admin/users).
Regarding the UI, there're two scenarios to check:
@latest_import
object is not present)As seen in the sandbox files, these two have slight differences.
Regarding the import behavior, it is expected that any CSV file with valid rows will create those records into the Repository. The operation is transactional (and won't be committed to the database)
in terms of database issues, but not in CSV format. This means that a CSV file with a few invalid rows can be partially imported into the database.
Final notes
There are a few improvements that can be still applied as soon as we do larger imports, such as:
Those have been added to the corresponding issue's description: #55. Let's leave it there just to get back to them on next iterations.