Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Document and export caching of API responses #23

Merged
merged 103 commits into from
Aug 30, 2022
Merged

Conversation

GregorDeCillia
Copy link
Contributor

@GregorDeCillia GregorDeCillia commented Jan 20, 2022

For quite some time there is a hidden feature that allows caching of API responses from the STATcube REST API. This is very useful for our internal web application and we will have to decide how we deal with this in the upcoming CRAN release.

Leaving it as a hidden feature might create a bad impression during reviews. Removing it would make it necessary to implement the caching logic elsewhere, which might be tricky. Therefore, it is probably best to document and export the behavior. Documentation is already available in ?sc_cache.

One problem with the current implementation is that the hashes are created via serialize() and therefore they are not reusable in different R versions. It would be very handy to use something like digest::digest() but adding another dependency package just for the hashes seems unnecessary. Maybe tools::md5sum() could be used in a different way to get a satisfying result.

TODOs

  • review reference documentation
  • think about creation of hashes
  • Add @export to make the caching available without environment variables
  • Remove @internal to include the man pages in the index page of the documentation
  • Maybe add something similar to od_cache_summary() which provides an overview about the cache contents. This would probably require some kind of cache_index.csv so we don't need to parse the cache entries.

add necessary modifications to NAMESPACE and
the reference index
@GregorDeCillia GregorDeCillia added the docs Improvements or additions to documentation label Jan 20, 2022
@GregorDeCillia GregorDeCillia added this to the Version 1.0 milestone Jan 20, 2022
@GregorDeCillia GregorDeCillia self-assigned this Jan 20, 2022
The constructor of the (internal) sc_table
class now uses totals=FALSE as a default.

This makes add_language() compatible
with sc_table_saved()
by default, sc_cache_enable() prints
instructions how keys can be set up
persistently using message()

to keep the log files of our internal
packages cleaner, these messages can
now be ommited using the verbose parameter
previsously, sc_key was always using the
environment variable STATCUBE_KEY to
return/store the key value

since we have multiple servers (see #25)
it is now possible to use

- STATCUBE_KEY_EXT for the external server
- STATCUBE_KEY_RED for the editing server
- STATCUBE_KEY_PROD for the production server

part of #25
and auto-detect the appropriate server in
cases where this can be inferred from the
request payload

part of #25
- namespace functions from utils and stats
- document all parameters for exported
  functions
- add missing parameter documentation for
  R6 method sc_table_class$initialize()
Add a parameter that allows to list all
datasets from the editing server instead of
the external one

part of #26
add a new "server" parameter that allows
to sitch between the external OGD server
and the OGD editing server. The cache
directory uses a subdirectory to cache
the responses from the editing server

TODO

- add documentation for new server
  parameter in od_table(), od_resource()
  and other affected functions
- Export functions that return base urls of
  the different servers. This is needed for
  a shiny app that is built on top of
  STATcubeR and links to the datasets
  via anchor tags
- extend the "resource management" vignette
  and mention how caching from the editing
  server is handled
- add parameter checking and throw errors
  if server is not "ext" or "red"

part of #26
for open data datasets, the FK-column
which can hold information about the
id of the parent element is now included
in the utput of the $field() method

part of #28
this caused issues in case time
variables were used with exactly
one classification level because of
speacial treatment

TODO: consider fixing this in
tabulate() instead since it kind
of makes sense to use a total code
here?
hopefully, this will make it easier to
get sensible defaults that work on all
platforms including
joke-operating-systems like MS-windows
possibly resolves "unexpected end of input"
error
turns out the API keys will have to be
requested and it is preferrable to
use the internal servers for the
STATcube GUI. Details about this are
documented in our internal confluence
and linked from the keys article
the preferences link only works when you
are already loggen in. Otherwise, there
will be a page "session expired"

mention this in the web-documentation
of STATcubeR
@GregorDeCillia
Copy link
Contributor Author

This branch now contains support for the external server of the API which will be released on 2022-08-31. To install STATcubeR from this branch, please use

remotes::install_github("statistikat/STATcubeR", ref = "export_sc_cache")

The version after the merge will be 0.4.0 and NEWS.md conains the most important changes compared to the current @master.

this server can now be used just like
any other server with

```r
# provide a key
sc_key_set("XXX", server = "test")
# send a request
sc_schema_catalogue(server = "test")
```
there is now a check to determine
wether STATcubeR is used inside the
firewall. If not, sc_browse() and
sc_browse_prefernces() now forwad to
the external STATcube and ensure that
the user is logged in when browsing
STATcube.

This commit also affects the $browse()
method of sc_table objects, which will
now determine the server from the json
request and open the correct deep-link
to the databases info page

possible TODOs:
- throw errors if server is not "ext"
  and is_stat() is FALSE
- use an environment variable in
  is_stat() rather than relying on
  Sys.info(). Otherwise, this the browse
  functions will only work on the two
  main R servers
use the english menu options to refer
to the GUI

use a prettier example to showcase the
datatree implementation of
print.sc_schema()
- explain what remaining and reset means
- link to the vendor docs about the
  endpoints table and schema
link to more related aricles
@GregorDeCillia GregorDeCillia marked this pull request as ready for review August 30, 2022 12:13
@GregorDeCillia GregorDeCillia merged commit bf9a035 into master Aug 30, 2022
@GregorDeCillia GregorDeCillia deleted the export_sc_cache branch August 30, 2022 13:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
docs Improvements or additions to documentation
Projects
None yet
1 participant