Can't save files in location

[eliocamp]

I've recently found your great package and wanted to test it and got this error:

library(ecmwfr)

query <- list(
  class="ei",
  dataset="interim",
  date="1979-01-01",
  expver="1",
  grid="2.5/2.5",
  levelist="200",
  levtype="pl",
  param="129.128",
  step="0",
  stream="oper",
  time="00:00:00/06:00:00/12:00:00/18:00:00",
  type="an",
  format= "netcdf",
  target="file.nc")

ecmwfr::wf_request("[email protected]", 
                   request = query, 
                   transfer = TRUE, 
                   path = getwd())

#> - staging data transfer at url endpoint or request id:
#>   https://api.ecmwf.int/v1/datasets/interim/requests/5c8fe53ebcd319dff7058b5a
#> 
#> - timeout set to 3.0 hours
#> - polling server for a data transfer
#> - writing data to disk ("/tmp/RtmpERehHV/ecmwfr_e491543096c.nc")
#> - moved temporary file to -> /home/elio/file.nc
#> - request purged from queue!
#> Warning message:
#> In file.rename(src, dst) :
#>   cannot rename file '/tmp/RtmpERehHV/ecmwfr_e491543096c.nc' to '/home/elio/file.nc', reason 'Invalid cross-device link'

The temporary file is in in there correctly downloaded, but "file.nc" is nowhere to be found. Changing the file.rename() call to file.copy() (see here) solves the issue but considering the potentially big files, it's probably not ideal.

I'm on elementaryOS 5 (based on Ubuntu 18.04)

[khufkens]

Will have a look at this!

[khufkens]

Ok, it seems that you might be trying to copy form a non encrypted temporary folder to an encrypted home folder as suggested here.

Could this be correct in your case?

[khufkens]

Ok, found that this might be a bug. Probably not on most systems but certainly on some. file.rename() re-assigns the location "tag" of a file (doesn't copy anything). Obviously if things are located on different drives / partitions or encrypted units this will not work (different partition tables and the likes). In most cases this doesn't come up, but if you assign a separate tmp partition or even physical drive, you run into this issue.

So although it will generate overhead the cautionary thing would be to use file.copy(), as file.rename() will not always work. Actually, better to trap the error with try() or tryCatch() and fall back to file.copy() in case of an error.

Thanks for flagging this!

[eliocamp]

I don't think is encrypted (I followed this) but my home is in another partition from / . So that must be the issue.

Would it be possible to download directly in the target file?

[khufkens]

Yes, there is. Basically wf_transfer() could take the final transfer location. But there was a reason why I didn't, but can't remember why.

CRAN rules in general are that you are not allowed to write by default to any user space. I probably did it to be on the safe side.

I also fear that incomplete downloads will leave behind gunk files. Cleaning those up is always a pain. While the tempdir() gets purged upon closing a session etc.

[khufkens]

Here I set a temp file name
https://github.com/khufkens/ecmwfr/blob/9dd901ed24216308682ba9cf3037bbfae980b8f6/R/wf_request.R#L145

Which technically could be the final destination path. But as said above, if the process breaks (time-out) or is exited due to crash / unclean exit. You are left with gunk in your "normal" spaces, not in a temporary location which gets purged from time to time.

[khufkens]

btw. this bug was fixed here: 9dd901e

You can pull the latest github version to get your downloads into your home directory.