Improve display of assertions on records pages

[nickdos]

Fixes required in biocache-hub:

• add i18n codes for all assertions to so lowercase values are shown
• remove the links and icons in ToC to missing and unchecked assertions
• Change code to use new sheet in Google sheet for the help popups (via config var dataQualityChecksUrl)
• Ensure help popups are working and at least showing a description and the raw code (uppercase version)
• ~~Add hide/show JS option and hide all assertions by default, as list is very long and not useful to most users. ~~

Fixes required in biocache-service:

• sort alphabetically within each section (warnings, passed)
• fix assertions appearing in both warnings and passed sections (duplicates)

[djtfmartin]

thanks to the i18n work @vjrj did, we can actually override i18n properties by putting updated messages_en.properties in any of the following paths:

        messageSource.setBasenames(
                "file:https:///var/opt/atlas/i18n/downloads-plugin/messages",
                "file:https:///opt/atlas/i18n/downloads-plugin/messages",
                "file:https:///var/opt/atlas/i18n/biocache-hubs/messages",
                "file:https:///opt/atlas/i18n/biocache-hubs/messages",
                "WEB-INF/grails-app/i18n/messages",
                "classpath:messages",
                "${application.config.biocache.baseUrl}/facets/i18n"
        )

[djtfmartin]

Fixed the majority this in 3.0.6 of biocache-service @nickdos i was able to change the i18n of the assertions without making any changes to ala-hub/biocache-hub .
I’ve fixed the case of all the assertion names.I made some changes to allow counts for:

• missing fields
• unchecked tests - using the related terms field on the OccurrenceIssue and ALAOccurrenceIssue.

I don't think we need to tackle any changes in biocache-hubs for pipelines v1.

[djtfmartin]

i think this has been verified by @peggynewman - moving to Done

[javier-molina]

? Help shows still errors to I think we need to keep this open while the spreadsheet placeholders are updated. I think we need to confirm the expand collapse requirement to hide all assertions.

[brucehyslop]

I've updated all the assertions codes to map to previous names where available and added the new assertions to the google sheet. biocache-test is configured to use the second sheet in the "Data Quality Checks with Pipelines", the first sheet is used by prod with the legacy assertion codes.

Note: the assertions names displayed as labels are still configured with the legacy names because biocache-hub uses the assertions name to generate the wiki link (eg. https://github.com/AtlasOfLivingAustralia/ala-dataquality/wiki/{assertion.name}). The wiki pages need to be renamed.

[javier-molina]

@brucehyslop I was under the impression help, even if it was only placeholders was going to be fixed. Is that fix still not in this release? see image below

Also related to the screenshot above, we still have both "Coordinate precision invalid" and "Coordinate precision not valid" probably one is ALA and the other GBIF. I was under the impression we are not checking this so the assertion didn't have to be there otherwise it would be passed for all records regardless. Happy to be corrected.

[brucehyslop]

@javier-molina what was this image captured?

There was a commit on Mon 26th May that resolved the duplicate "Coordinate precision invalid" assertion. This was merged on Tue 27th and should have been deployed shortly afterwards.

As for the help tooltips, I tested all of them last week and didn't find any that where missing. My only thought is that the cache of the google sheet that contains the data may not have been refreshed on every ala-hub node and the load balancer was directing me to an updated node when I test.
Maybe we need to do a restart of all nodes to ensure the caches are flushed.

[javier-molina]

Hi @brucehyslop

I just tested now. Help is showing properly now.

Duplicate precision invalid/not valid still showing though.

[brucehyslop]

@javier-molina I can't seem replicate this. Can you provide the occurrence record id

[javier-molina]

@brucehyslop I should've added this ages ago
It is https://biocache-test.ala.org.au/occurrences/03068f05-d4ff-41b7-aef2-cec84bee75f5

[javier-molina]

And "Missing coordinate precision " has to go too please.

[brucehyslop]

Removing the MISSING_COORDINATEPRECISION ALAOccurrenceIssue has re-indexed all the Assertion codes. Which requires an update to the "Data Quality Checks" google sheet.

The update has been made however the until the deployment of the latest biocache-service snapshot release and flushing of the biocache-hubs caches there may be an issue with the tooltips being out of sync with the assertion. Note: this will only effect ALAOccurrenceIssues

[brucehyslop]

Also removed duplicate assertion GEOREFERENCED_DATE_UNLIKELY that was defined in both the GBIF OccurrenceIssue and ALAOccurrenceIssue

[alexhuang091]

I still saw 2 entries on https://biocache-test.ala.org.au/occurrences/03068f05-d4ff-41b7-aef2-cec84bee75f5

[brucehyslop]

changed label for the assertion COORDINATE_PRECISION_UNCERTAINTY_MISMATCH previously use the label of the deprecated assertion coordinatePrecisionMismatch

[RobinaSanderson]

Tested by

1. search for all records
2. select optio to display records with spatial issues
3. view a record in the search results
4. scroll to the Data quality checks
5. Click on the Info icon next to the data quality issues found

Currently displays an error
Error: info not found

[brucehyslop]

The cause broken tooltips is biocache-hub is returning the old assertions codes (https://biocache-dq-test.ala.org.au/dataQuality/allCodes) however the assertions codes are retrieved from biocache-service (https://nci3-biocache-service.ala.org.au/ws/assertions/codes) which has the correct codes.

Possible a cache issue however the cache should not contain pre-pipeline assertion codes, a rebuild and deployment to NCI3 should not have maintained the cache.

[brucehyslop]

Has there been a change to the deployment or config? This was working last week (Friday).

[brucehyslop]

Found the issue.
The https://biocache-dq-test.ala.org.au/dataQuality/allCodes uses the Data Quality checks google sheet. The ansible config was was not pushed to the repo and was clobbered with the last deployment to test.

https://github.com/AtlasOfLivingAustralia/ansible-inventories/commit/f7b2641122e08b27aa107b56cf4e8b58af3f8659

[RobinaSanderson]

There was an issue where all assertions appeared to have wiki links whether there was actual pages created for the assertions with that name or not. This meant, if a page hadn't been created, any user clicking on the link would be presented with a wiki creation page and asked to create their own.

To fix this I edited the data quality checks google doc spreadsheet on the "Data Quality Checks - Pipelines" tab to remove the text "wiki" in the wiki link column for all those assertions that don't yet have a page. I then refreshed the cache at https://biocache-dq-test.ala.org.au/admin

This should have resulted in the info for assertions with a page displaying a link, and the info for pages without links not displaying a link. See this image from Prod.

Actual result is an error message on all links whether a page exists or not. I will recheck on Monday morning, but it looks like something else is broken.

[javier-molina]

@nickdos I remember you fixed the inventory for AVH in #455 that had the same issue.

Could you please check if the same issue is going on in test (biocache-dq-test)?

[nickdos]

@javier-molina Yes, biocache-dq-test is missing the added param that was also missing in AVH. I'll check ansible and make sure its saved there before fixing the dq-test site.

[nickdos]

Fixed in dq-test:

[RobinaSanderson]

Tested by:

1. opening https://biocache-dq-test.ala.org.au/occurrences/d4d76e97-4eaa-424b-9e8e-e57937454b32
2. Opening Show/Hide passed properties
3. clicking on the info symbol next to the check

Expected results

1. Where a github wiki page doesn't exist no link will be shown
2. Where a wiki page does exist a link will show and clicking it will take you to the page
3. Where a n assertion has been renamed the link will be to the wiki page for the new name

Actual results

1. is OK - example is MULTIMEDIA_DATE_INVALID
2. Assertions in biocache pre pipelines with pages display links. Assertions new in pipelines don't display links even where they exist example is CONTINENT_COUNTRY_MISMATCH
3. Assertions with new names are linked to the old assertion pages, even is a new page exists - BADLY_FORMED_BASIS_OF_RECORD

Wondering if the code is referencing the right worksheet in the DataQuality Checks google doc? The correct sheet with all new assertion names is https://docs.google.com/spreadsheets/d/1tAfQUsIzUNfa6Tn5Ezq2cfbMVB0QeXIyp7N0We2lBAY/edit?hl=en&hl=en&hl=en#gid=820855861 Data Quality Checks - Pipelines

[RobinaSanderson]

Retested
I can confirm results are now as expected.
@javier-molina I think this can be moved to done.