index on national EPBC conservation status

[M-Nicholls]

Add an index on EPBC status from the national list (dr656), similar to the the state conservation index already implemented.

potentially add a parameter to the biocache tool "update-conservation-data" command (see below), to select state conservation or national EPBC

Command: update-conservation-data
Description: Load conservation data from sources (e.g. list tool)
Usage: update-conservation-data
No arguments required for tool

[M-Nicholls]

High priority - required to facet records to produce reports for State of the Environment Reporting

[adam-collins]

countryConservation exists in cassandra the same as stateConservation. It is produced by matching list region with processed country. e.g. http:https://biocache.ala.org.au/ws/occurrences/517da990-31e2-4beb-a072-953f4d3e25b0

Adding raw_country_conservation and country_conservation to the index is reasonable if this is more than a one-off report.

For a one-off reporting I would suggest creating an authorised=true list for each segment of dr656 as required. After weekly indexing results can be filtered with each new dr, e.g. q=species_list_uid:dr656&fq=country:Australia. After use these segments can be authorised=false or deleted to remove from the next index.

It appears update-conservation-data is currently disabled. This means running it will not update threatened and authorised lists for use in stateConservation, countryConservation or globalConservation cassandra fields. update-conservation-data needs fixing.

[M-Nicholls]

We should add raw_country_conservation (comes from the "sourceStatus" field) and country_conservation ("status") to the index, being able to access records by conservation status has been requested in the past. The "sourceStatus" field is the actual EPBC value and should be used as the facet if it's added to the UI.

[adam-collins]

Added raw_country_conservation and country conservation to indexing.

Updated update-conservation-data. This will reset state and country taxon conservation information to match threatened and authoritative lists that have "sourceStatus" and "status". Reprocessing is required to update occurrences.

50aa78c and 8d8118c

[adam-collins]

Now available in the index.
http:https://biocache.ala.org.au/occurrences/search?q=country_conservation:*
http:https://biocache.ala.org.au/occurrences/search?q=raw_country_conservation:*

[M-Nicholls]

thanks Adam, I missed it when this went in but it would have been impossible to the SoE reporting without it.

[nickdos]

Re-opening as the index is showing zero records for both country conservation and raw country conservation fields at the moment.

[nickdos]

One possible cause is the EPBC list was updated on 2020-06-04. This edit may've introduced content (chars) that are causing the indexing of the fields to fail (guess).

[patkyn]

The Update Conservation Data job updates taxon table with conservation status. This has not been run for a while. After running the job and doing a Complete Reprocess and Indexing, the https://biocache.ala.org.au/occurrences/search?q=country_conservation%3A* is showing 1,234,081 records. However, because dr656 does not have sourceStatus the https://biocache.ala.org.au/occurrences/search?q=raw_country_conservation%3A* is not showing any records

[nickdos]

Do we need to remove raw_country_conservation from index & docs? I'm not sure it makes sense as we're not trying to map 8+ status vocabs to a single vocab, like we do for states conservation...