Possibly breaking changes

• CTGOV has retired on 2024-06-25 the classic website and API used by ctrdata since 2015. To support users, ctrdata now automatically translates and redirects queries to the current website. This helps with automatically updating previously loaded queries (ctrLoadQueryIntoDb(querytoupdate = <n>)), manually migrating queries and reproducible work on clinical trials information. This new functionality in ctrdata translates a user's search query URL from the classic website into a query for the current CTGOV website, for all search parameters. Since the structure and format of data differs between data retrieved from the current API and previously retrieved from the classic API, ctrdata will continue to identify the current API as register = "CTGOV2", to support the analysis stage. In addition, ctrdata documentation continues to include examples of analyses with CTGOV data, as this may have been downloaded earlier.

• CTIS has been relaunched on 2024-06-17, and ctrdata has been fully updated to it. At the moment, CTIS provides basic searches and no search query URL. To support users, ctrdata includes an updated script that extracts a user's search parameters from the register search page to the clipboard and into the browser URL bar. In addition, the script triggers a search for trials when opening such a query URL, see https://rfhb.github.io/ctrdata/#id_2-script-to-automatically-copy-users-query-from-web-browser. File names of CTIS documents start now with the document type, e.g. SbjctInfaICF - L1 SIS and ICF NL for publication.pdf, since the prefix (document source) introduced in version 1.18.0 is no more applicable.

Improvements

• Reduced size of demo database in package, addressing CRAN notes
• Adapted ctrFindActiveSubstanceSynonyms() to CTGOV2 API; this is now based on terms used in studies
• Updated vignette and added inspecting a specific trial
• Updated register information, adding caveats and issues
• Updated estimated download sizes and user information
• Avoided duplicate data and file retrievals
• New attribute ctrdata-collection for data frames with trial information; same content as attribute ctrdata-table, which will be removed by end 2024
• Split utils.R into files for functions and fields

Bug fixes

• Correct partial date brackets for CTGOV2
• Disable HTTP/2 multiplexing for CTIS

Possibly breaking change

• File names for documents downloaded from CTIS now include the document type and use different separators (e.g., "parts2auth - SbjctInfaICF - ..." as abbreviation of "Subject information and informed consent form (for publication)", was previously "parts2auth_...")

Improvements

• Add retrieval of historic versions of trial records from CTGOV2 (automatically retrieved from CTIS, not available for other registers)
• Added typing of newly appearing fields from CTIS
• Increase testing coverage to 93.7% locally
• Added missing CI for the combination of DuckDB and CTGOV2

Bug fixes

• Correct typing certain fields as (lists of) integers
• Handle marginal case in dbGetFieldsIntoDf()
• Changed unlink of intermediate files
• Minor vignettes corrections

Bug fixes

• Switch sequence of API endpoints used with CTIS
• Correct handling multiple public events with CTIS
• Re-use CTIS downloads in a given session

Improvements

• Additional CTIS field types and possibly documents (associated clinical trials)
• Use ctId instead of id as CTIS top-level field (a further synonym used in the API is clinicalTrialId)

Bug fixes

• Added handling of unexpected Unicode in protocol-related data from one trial in EUCTR
• Improved predicted download sizes and times for EUCTR
• Return with message if CTIS query did not result in any trials

Possibly breaking changes

• Reimplemented dbGetFieldsIntoDf() to accelerate and have more predictable, simplified returns, in particular for nested fields; also attempts to recursively expand simply nested data into additional columns in the returned data frame
• Reimplemented dbFindFields() to accelerate; both based on improved nodbi::docdb_query()
• dbFindFields() now digests a sample of records to quickly find fields, or all records if sample = FALSE but taking increasing time with increasing number of records
• If using nodbi::scr_postgres(), parameter fields of dbGetFieldsIntoDf() is limited to less than 50 fields; a message flags for any backend potential compatibility issues, suggesting to use parent fields, e.g., a.b instead of c("a.b.c.d", "a.b.c.e")
• Parameter stopifnodata of dbGetFieldsIntoDf() is no more needed and deprecated
• Reimplemented typing fields to speed up and to simplify

Improvements

• Register data are re-used and not downloaded again in an interactive session (that is, the same temporary folder is now re-used throughout a user's session)
• Temporary folder can be set by users with options(ctrdata.tempdir = "<user_specified_folder>")
• Inform MS Windows users if cygwin was found so that they may chose to delete it
• Many fields added for typing e.g. as date in dbGetFieldsIntoDf()

Bug fixes

• Adapted and corrected information loading to newly available data in CTIS
• Corrected escaping, and back-conversion, of characters in JSON from CTIS

Possibly breaking changes

XML files are converted slightly differently

• EUCTR result-related information in attributes: e.g. new: {"id":"PostAssignmentPeriod-46349"}, was: {"@attributes":{"id":"PostAssignmentPeriod-46349"}}
• Consequently, it should work to just delete @attributes from field names such as dbGetFieldsIntoDf("clinical_results.baseline.analyzed_list.analyzed.count_list.count.@attributes.value", db)
• EUCTR protocol-related information although no differences were found yet
• CTGOV attributes of bare values remain not included in the resulting NDJSON (e.g., some but not all records have <start_date type="Actual">March 15, 2004</start_date>, and this is converted to {"start_date":"March 15, 2004"})

EUCTR: some renaming to harmonise EU and 3rd country trial fields

• new: e83_single_site_trial, was (EU trials): e83_the_trial_involves_single_site_in_the_member_state_concerned
• new: e83_single_site_trial, was (3rd country trials): e83_will_this_trial_be_conducted_ at_a_single_site_globally
• new: e863_trial_sites_planned_in, was (EU trials): e863_specify_the_regions_in_which_trial_sites_are_planned
• new: e863_trial_sites_planned_in, was (3rd country trials): e863_specify_the_countries_outside_of_the_eea_in_which_trial_sites_are_planned
• new: e84_multiple_sites_in_member_state, was (EU trials): e84_the_trial_involves_multiple_sites_in_the_member_state_concerned
• new: e840_multiple_sites_globally, was (3rd country trials): e84_will_this_trial_be_conducted_at_multiple_sites_globally

See also #26 (comment)

Bug fixes

• corrected batch iterations over CTIS trials accommodating unclear totalSize response
• corrected translation of some fields from the browser URL to the API call for CTGOV2 (closes #32)
• corrected minimum curl version to 5.1.0 (closes #31)
• handled errors when saving EUCTR results (e.g., too long file path name, closes #30 and #28)

Improvements

Major

• No external tools required anymore (Cygwin, perl, cat, sed, php functionality for transforming text, XML and NDJSON replaced by Javascript using R package V8); addresses personally communicated concerns and faciliates use of package ctrdata in more environments (e.g., #26); consequently, this might be a breaking change for analysing certain fields, see above which fields are affected.

Other

• added results summary download for CTIS
• added documents download for ISRCTN
• factored out document download function
• ensure dbFindFields() returns fields for EU and 3rd country trials in EUCTR (addresses #26)
• changed order of importing from CTIS into database, improved speed
• better checked data downloads, and repeat them where necessary
• factored out temporary folder creation
• added using options(ctrdata.tempdir = ...) if set
• removed dfListExtractKey(), long deprecated
• removed dfMergeTwoVariablesRelevel(), long deprecated
• reorganised code file layout

• fix handling as utf8 upstream multi-language strings from CTIS
• correct creating lists for downloading documents for CTIS
• adding missing endpoints for CTIS found with increasing amount of data, e.g. publicevents.temporaryHaltList.details for 2022-501559-99-00

• correct LaTeX documentation resulting in CRAN error
• correct parts of downloading from CTIS, including file name sanitisation

• added CTGOV REST API 2.0.0.-test as new register with identifier CTGOV2
• handle CTGOV classic interface as register CTGOV
• improved ctrdataURLcopier.js to only rewrite searches and views from CTIS
• mangle CTIS: change partIIInfo object into array, adding a new partIIIinfoKey so that '{"partIIInfo": "<int>": {...}, "<int>": {...}}' becomes '{"partIIInfo": [{"partIIIinfoKey": <int>, ...}, {"partIIIinfoKey": <int>, ...}]}')
• correct dbGetFieldsIntoDf() for specific lists

• fix dbFindIdsUniqueTrials() for single-record register contents
• expand number of CTIS variables that are typed as date
• dfMergeVariablesRelevel() superseeds dfMergeTwoVariablesRelevel()