This repository contains metadata transformations, validations, and sample data (fixtures) used by PA Digital aggregation processes. The validations and transformations are written in Schematron and XSL. While built for the PA Digital aggregator, the validations and transformations are created to be platform-independent (within realm of XML technologies).
Tasks associated with PA Digital aggregation, including transformations and validations, are managed and executed with the open source platform Apache Airflow using institution-specific automated workflows. XSLT and Schematron files described here are called using Airflow variables, which are maintained as a JSON file (variables.json) in funcake_dags and imported manually into Airflow. For more information, see About the PA Digital Aggregator.
The same filename is used across different file types (e.g., cdm_temple.xsl
is used to transform the sample file cdm_temple.xml
and is tested using cdm_temple.xspec
). Except where an institution has a local or unnamed repository, the first part of the filename indicates which repository it is used for and the next is the ID of the institution we use locally. For our use in Airflow, dag_ids maintained in funcake_dags should match the institution code used in filenames and the lookup table in aggregator_mdx, minus the prepended repo code. For example, the corresponding dag_id for "cdm_temple.xsl" should be "temple".
To run one of the institution- or repository-specific XSLT, you need to have all 3 files in the directory structure provided here.
This documentation and the infrastructure described here was originally written and created by Christina Harlow and later updated to reflect evolving practice and use.
The XSLT (all version 2) is written to process XML metadata harvested via OAI-PMH as well as XML or CSV that are unavailable via OAI-PMH but stored in PA Digital's private data repository.
Transformations are written at three levels:
-
institution- or repository-specific XSLT (e.g.,
transforms/cdm_temple.xsl
): these are the files you actually run against the desired XML files and generally include mappings to crosswalk metadata whose location or presentation varies depending on repository and institution. These include digital object and thumbnail preview URLs; IIIF URLs; data provider, intermediate provider, and collection names; locally generated identifiers; and any mapping that should override the base crosswalk (next). -
base crosswalk XSLT (e.g.,
transforms/oai_base_crosswalk.xsl
): this file is imported above and contains XML node templates that are reused across collections (for example, mapping incoming dc:title to PA Digital's dcterms:title, dc:creator to dcterms:creator, etc.). The OAI-PMH base crosswalks delimit on semicolon for all but the following fields where semicolon is sometimes used as punctuation: description, title, publisher, extent, relation, and rights. There are also base crosswalks to process CSV or XML stored in PA Digital's private data repository (csv_generic_semic.xsl
, which delimits on;
andcsv_generic_pipe.xsl
, which delimits on|
). Base crosswalks should not be modified unless a change is desired for all processed collections. If an override of the base crosswalk is required, it should be done in the institution- or repository-specific XSLT described above (seedcterms:isPartOf
mapping incdm_temple.xsl
for an example of a crosswalk override; thepriority
attribute should be used to avoid ambiguous rule matching). All base crosswalks import the remediation XSLT (next). -
remediation-specific XSLT (
tranforms/remediations/*.xsl
): these files have normalization & enhancements.remediations/lookups.xsl
has lookup parameters used by the above templates to normalize string values against a variety of vocabularies, including DCMI Types, the DPLA-recommended Getty AAT subset, Lexvo Language look-ups, month abbreviations, etc. It is often used in institution or repository crosswalks to generate data provider names by a base URL, collection names by setSpec values, institutions codes for use in our local identifiers, etc. An example of using these lookup params can be found insamvera_shi.xsl
(seeidentifier
anddataProvider
templates).
Our validation scripts are written in Schematron, which uses XPath to test for different elements and patterns within the metadata. These files are found in the /validations
folder. Validation scripts run at two points in the aggregation workflow - the first is immediately after ingest of metadata, and the second runs after the transform tasks. Within these steps, two separate scripts run concurrently. One serves as a filter; any invalid records as defined by the script do not proceed to the next task. Along with a human-readable error note such as, “There must be a title,” invalid records are listed in a CSV that can easily be passed on to data providers for review and remediation. The other script serves as a reporting tool; 'invalid' records do continue on to the next task. This creates a list of records that match specified XPath selections for assessment purposes. For example, validations/padigital_missing_thumbnailURL.sch
tests for records that lack thumbnail URLs, allowing us to provide CSV reports to data providers without preventing records from moving to the next task.
In order to pass validation, the test must resolve to TRUE. If it resolves to FALSE, it is invalid. The tests should check for what should be present, rather than what should not. In the example below, an object URL in edm:isShownAt must begin with 'http.' If it does not, the validation fails with the following response: 'edm:isShownAt must contain a URL.'
<pattern id="ItemURLElementPattern">
<title>Additional Trackback URL Requirements</title>
<rule context="oai_dc:dc/edm:isShownAt">
<assert test="normalize-space(.)" id="ItemURL1" role="error">The trackback URL must contain text</assert>
<assert test="starts-with(normalize-space(.),'http')" id="ItemURL2" role="error">edm:isShownAt must contain a URL</assert>
</rule>
</pattern>
You can install and run the following commands locally without Docker & Docker-Compose; you'll need to follow the installation instructions for your system (installing a Saxon processer; Java; and XSpec. We recommend installing Docker & Docker-Compose however, and our make
commands only work in the Docker context.
Before running any of the below make
commands, you'll want to make sure Docker is running on your system and docker-compose
is in your local PATH. To verify this, you can run the following:
$ docker -v
Docker version 19.03.4, build 9013bf583a
$ docker-compose -v
docker-compose version 1.24.0, build 0aa59064
You just want both commands to return a version; the version and build numbers may vary from above, and that's okay.
Additionally, you'll need this repository, aggregator_mdx
, available locally, and you need to be in the top level of this repository. If you do not have this repository locally, you can run:
$ git clone https://github.com/tulibraries/aggregator_mdx.git
$ cd aggregator_mdx
To run XSLT against an XML document:
- Both the XSLT document(s) & the XML document need to be in this repository, e.g., in the
aggregator_mdx
directory or subdirectories. - Run the following command to build a Saxon image with your local directories available to it, then to run the
fixtures/sample.xml
XML document against thetransforms/sample.xsl
XSLT & output the result tosample-result.xml
:
$ make s=fixtures/sample.xml xsl=transforms/sample.xsl o=sample-result.xml saxon
All three arguments are required:
s
: the source XML document you want to transform;xsl
: the XSLT document you want to apply to the XML;o
: the output document where the resulting XML is written.
Unit tests for the XSLT and Schematron documents are written using XSpec. See their GitHub wiki for further examples and details on how to write and use these tests (for example, the XSpec tutorial folder in XSpec).
To add tests:
- Create an xspec file within
tests
directory, putting XSLT test files intests/xslt/
, and Schematron test files intests/schematron/
. The test file should be the same exact name of the XSLT or Schematron file you're testing (e.g.,tests/schematron/padigital_reqd_fields.xspec
contains the tests forvalidations/padigital_reqd_fields.sch
). - Add the
x:description
root element, adding namespaces you'll be using in sample data & either aschematron
or astylesheet
pointing to the .sch or .xsl file you're testing, e.g.:
<?xml version="1.0" encoding="UTF-8"?>
<x:description xmlns:x="http:https://www.jenitennison.com/xslt/xspec"
schematron="../../validations/padigital_reqd_fields.sch">
</x:description>
or
<x:description xmlns:x="http:https://www.jenitennison.com/xslt/xspec"
xmlns:xsl="http:https://www.w3.org/1999/XSL/Transform"
xmlns:xsi="http:https://www.w3.org/2001/XMLSchema-instance"
xmlns:dc="http:https://purl.org/dc/elements/1.1/"
xmlns:dcterms="http:https://purl.org/dc/terms/"
xmlns:dpla="http:https://dp.la/about/map/"
xmlns:edm="http:https://www.europeana.eu/schemas/edm/"
xmlns:oclcdc="http:https://worldcat.org/xmlschemas/oclcdc-1.0/"
xmlns:oclcterms="http:https://purl.org/oclc/terms/"
xmlns:oai="http:https://www.openarchives.org/OAI/2.0/"
xmlns:oai_dc='http:https://www.openarchives.org/OAI/2.0/oai_dc/'
xmlns:oclc="http:https://purl.org/oclc/terms/"
xmlns:oai_qdc="http:https://worldcat.org/xmlschemas/qdc-1.0/"
xmlns:schema="http:https://schema.org"
xmlns:svcs="http:https://rdfs.org/sioc/services"
stylesheet="../../transforms/dplah.xsl">
</x:description>
- Within the
x:description
parent node, write a test scenario (x:scenario
) to test each case per validation rule, XSL matched template, or XSL named template. You can nestx:scenario
elements as makes sense for what you are testing (e.g., for the schematron tests, you could add a top-level scenario per pattern, then a child scenario within each for both checking valid XML passes and invalid XML returns the expected errors). - Each XSpec test has sample XML ('fixtures') that is passed in and the result checked. This data is passed via
x:context
within each scenario. You can either put in the fixture XML directly in thex:content
node in eachx:scenario
, or you can havex:context
reference fixture files infixtures
via anhref
attribute and relative path to the file in the repository.
When ready to check your test suite locally, you can run the tests as described below.
To run just the XSLT Unit tests (e.g. every .xspec
file within tests/xslt/
):
$ make test-xslt
To run just the Schematron Unit tests (e.g. every .xspec
file within tests/schematron/
):
$ make test-sch
To run all Unit tests (e.g. every .xspec
file within tests/schematron/
and within tests/xslt/
):
$ make test
All of the above generate test reports in the relevant test directory, e.g. tests/schematron/xspec
or tests/schematron/xspec
. These are HTML and XML versions of the output, which can be helpful to review if debugging a failing test (just open the directory in your web browser via a local filepath). These are always ignored by git, so never checked into GitHub.
Alternatively, you can run individual unit tests from within Oxygen. See documentation here. Please note that this only tests a single XSpec file. It is handy to troubleshoot and check tests quickly, but since transforms are often run against multiple sample files, the integrity of the repository depends on all tests running sucessfully. The Docker commands described above test all of the XSpecs in a folder, and is far more reliable.
CircleCI manages running CI/CD for this application. It does the following, using the same Docker images we use locally:
- build the Docker image with the repository files from that commit copied over;
- run a version of
make test
with a command to fail ifError running the test suite
orError compiling the test suite
appears in the test output. - run a coverage check that for every
.sch
file invalidations/
and.xsl
intransforms/
has a test file[same filename].xspec
intests/
. CI will fail if this coverage is less than 70%.
The transformation and validation tasks in Airflow pull the main
branch of this repository; so deployment is via successful CI and merging of PRs into main
.