Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add support for SDMX #27

Open
GregorDeCillia opened this issue Mar 23, 2022 · 1 comment
Open

Add support for SDMX #27

GregorDeCillia opened this issue Mar 23, 2022 · 1 comment
Labels
Milestone

Comments

@GregorDeCillia
Copy link
Contributor

GregorDeCillia commented Mar 23, 2022

It would be very useful if {STATcubeR} could support "SDMX archives" which are generated from STATcube. sdmx archives consist of a metadata component called the "structure definition" and a data part which contains the actual cell values. In order to support that, we would need to add parsers for the xml-based data format.

The generated archives are more or less compatible with the CRAN package rsdmx: https://cran.r-project.org/package=rsdmx, which could be used as a starting point to develop parsers.

Possible usuage: parser function sdmx_table() which generates an object of class sc_data (the parent class for OGD and STATcube-API datasets)

x <- STATcubeR::sdmx_table("path/to/sdmx_archive.zip")
class(x)
#> [2] "sdmx_table" "sc_data" "R6"

There are several advantages of the sdmx format compared to the API

  • The structure definition contains information about hierarchical classifications, which are not available via the API
  • The download option "sdmx archive" is available even if STATcube is used as a "guest user"
  • SDMX is used by other SuperSTAR products such as SuperCROSS

The last point is probably the most compelling one since a direct interface to SuperCROSS would be very helpful for the internal workflows of statistics austria

@GregorDeCillia GregorDeCillia added this to the Nice to have milestone Mar 23, 2022
@GregorDeCillia
Copy link
Contributor Author

Screenshot: SDMX downloads are available for unregistered users on the external STATcube instance

image

GregorDeCillia added a commit that referenced this issue Mar 29, 2023
this is the first step to resolving
#27 by adding a function that creates
sc_table() like objects based on sdmx
archives

The sdmx format contains all metadata
that is necessary for STATcubeR to reuse
the existing $tabulate() workflow and this
first version already provides support for
various features via the base class (sc_data)

- $tabulate() to aggregate data
- $total_codes() to set/unset total codes
- $recoder to recode datasets (change labels)
  change codes, toggle visibility of
  elements, reorder elements, etc.
- importing german and english labels
  simultaniously (both languages are included
  in a zip download) and allowing to swhitch
  between them using $language<-().

New features
- sdmx arcives provide a $parent column
  in the $fields() table which are used
  to represent hierarchical classifications.
  Previously, this was only possible with
  od_table()

There are still some improvements. See
the issue #27 for more details

- properly parse time variables -
  currently they are treated as generic
  categories.
- parse element annotations (detailed
  descriptions for classification
  elements) and add them to
  $field()$de_desc just like with
  OGD dataset
- parse value annotations (see #39)
- provide a print/fromat method
- add a reasonable logic for total
  codes that takes the parent codes into
  account
- fill meta$measures$fun and
  $meta$measures$precision based on
  the sdmx metadata
- modify very long codes which use
  the @-symbol (probably for escapes)
- extend documentation
- possibly check SuperCROSS compability
@GregorDeCillia GregorDeCillia linked a pull request Apr 6, 2023 that will close this issue
5 tasks
@GregorDeCillia GregorDeCillia pinned this issue Apr 6, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant