Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

bm_raster returns NULL #13

Open
simon-smart88 opened this issue Jun 14, 2024 · 7 comments
Open

bm_raster returns NULL #13

simon-smart88 opened this issue Jun 14, 2024 · 7 comments

Comments

@simon-smart88
Copy link

This has been the case for a month or so now. I've updated to the dev version but the problem persists. It seems unrelated to the sf object that I pass.

library(sf)

square_coords <- matrix(c(
  -1.1, 50.5,  
  0.9,  50.5, 
  0.9,  52.5,  
  -1.1, 52.5,  
  -1.1, 50.5   
), ncol = 2, byrow = TRUE)

square_polygon <- st_polygon(list(square_coords))
square_sf <- st_sfc(square_polygon, crs = 4326)  
sf_object <- st_sf(geometry = square_sf)

blackmarbler::bm_raster(roi_sf = sf_object,
                        product_id = "VNP46A4",
                        date = "2021",
                        bearer = Sys.getenv("NASA_bearer"),
                        quiet = FALSE)

Produces:

Processing 2 nighttime light tiles
Processing: VNP46A4.A2021001.h17v03.001.2022094115448.h5
  |========================================================================================================================| 100%
Downloading: 4.1 kB     NULL
@ramarty
Copy link
Contributor

ramarty commented Jun 14, 2024

@simon-smart88 Thanks for flagging the issue. Was testing now, and there seem to be some issues on the NASA LAADS archive. Will keep checking to see if the problem is with the code or with the archive that the code is trying to access — and at the very least aim to include some more helpful error messages when there seems to be issues with the LAADS archive.

@firatgundem
Copy link

I think the problem still exists. Any improvements? Below my codes:

roi_sf <- gadm(country = "GHA", level=1, path = tempdir())

Daily data: raster for February 5, 2021

r_20210205 <- bm_raster(roi_sf = roi_sf,
product_id = "VNP46A2",
date = "2021-02-05",
bearer = bearer)
r_20210205
NULL

@firatgundem
Copy link

I think no one is checking this page. Is there anyone who can help? Why does bm_raster create a null data set?

@ramarty
Copy link
Contributor

ramarty commented Aug 12, 2024

@firatgundem Sorry for the delay here! Does the dev version work for you?

install.packages("devtools")
devtools::install_github("worldbank/blackmarbler")

Sometimes there's issues as the NASA LAADs archive is down which will cause NULL to be returned - I'm working on having the package return a message when this is the case, but one way to see if that's the source of the error is if you can manually download a file:

https://ladsweb.modaps.eosdis.nasa.gov/archive/allData/5000/VNP46A4/2012/001/

Another issues could be the bearer token expired, so could be worth trying to regenerate that.

@firatgundem
Copy link

Image

Yes, I tried devtools as well but got the same thing. I have been trying to obtain this blackmarble dataset for more than 1 week and I could only get the null data. Could you please help me?

@ramarty
Copy link
Contributor

ramarty commented Aug 12, 2024

Ah thanks for this!

1/ Could be worth making sure all the packages it depends on are up to date?

readr,
hdf5r,
dplyr,
purrr,
lubridate,
tidyr,
terra,
sf,
exactextractr,
stringr,
httr

2/ Are you on a mac or pc?

@firatgundem
Copy link

I use pc and my packages are up-to-date. This is the latest error that I got:

black_marble_data = bm_raster(roi_sf = shapefile, # this specifies the region of interest

  •                            product_id = "VNP46A3", # this specifies the monthly luminosity data
    
  •                            date = seq.Date(from = ymd("2021-10-01"), to = ymd("2021-12-01"), by = "month"), # this specifies the date range
    
  •                            bearer = bearer, # this is the API bearer for NASA LAADS DAAC from Step 3
    
  •                            variable = c("AllAngle_Composite_Snow_Free"), # this selects the layer we're interested in i.e. satellite angle and snow-free observations
    
  •                            quality_flag_rm = c(255, 2)) # this drops bad quality observations and coverts them to NA values)
    

Processing 1 nighttime light tiles
Processing: VNP46A3.A2021274.h33v12.001.2021321133308.h5
|===============================================| 100%
Error in downloading data
No encoding supplied: defaulting to UTF-8.

<title>504 Gateway Time-out</title>

504 Gateway Time-out


nginx

Error in downloading data; bearer token likely invalid. Try regenerating the bearer token; please see this link for instructions to obtain a bearer token: https://github.com/worldbank/blackmarbler?tab=readme-ov-file#bearer-token-
Processing 1 nighttime light tiles
Processing: VNP46A3.A2021305.h33v12.001.2021343154743.h5
|===============================================| 100%
Warning: PROJ: proj_create_from_database: C:\Program Files\PostgreSQL\14\share\contrib\postgis-3.3\proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation. (GDAL error 1)
Processing 1 nighttime light tiles
Processing: VNP46A3.A2021335.h33v12.001.2022008110827.h5
|===============================================| 100%
Warning: PROJ: proj_create_from_database: C:\Program Files\PostgreSQL\14\share\contrib\postgis-3.3\proj\proj.db contains DATABASE.LAYOUT.VERSION.MINOR = 0 whereas a number >= 2 is expected. It comes from another PROJ installation. (GDAL error 1)

Storing the layer names from the original raster stack

original_layer_names = names(black_marble_data)

Imputing missing values in the raster data

imputed_data = approxNA(black_marble_data, method="linear", rule=2, f=0) #addressing the NA values by linear interpolation
Error: unable to find an inherited method for function ‘approxNA’ for signature ‘x = "NULL"’

It is also interesting to see these. Although it says the bearer token is likely invalid, I have just renewed it. How can it be invalid?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In progress
Development

No branches or pull requests

3 participants