Help: Understranding StackRaster

[ThomasAuriel]

I'm using this line of code to load data:

ds_stack = RasterStack(file; name=variables, lazy=true, dropband=false, missingval=NaN, source=:netcdf)

then I try to extract ds from the ds_stack with this:

## Store data
            for variable in variables

                    ds = ds_stack[variable]
                    name = string(ds.name)
                    # Define folder to store variable
                    folderpath, __ = settings.getFolderpath(name)

                    # If the folder does not exist, create it
                    FileUtils.createFolder(folderpath)

                    for time in lookup(ds, Ti)

                        try
                            ## Store element
                            subDs = ds[Ti=At([time])]
                            write(filepath, subDs, force=true)
                            println("done")
                        catch ex
                            println("Error during processing: $(ex)")
                        end

                    end
            end

It's working with some file but not with other. I don't understand the difference between them.
I opened the files with another app but I don't saw the difference (possible it is hiden by the app)

It's working with:

> ds_stack
RasterStack{Rasters.FileStack{Rasters.NCDsource, (:t2m,), String, Tuple{Union}, Tuple{Tuple{Int64, Int64, Int64}}, Tuple{DiskArrays.GridChunks{3}}, Tuple{DiskArrays.Unchunked}}, Tuple{X{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDsource, Dict{String, Any}}, EPSG, EPSG, X{Colon}}}, Y{Mapped{Float32, Vector{Float32}, DimensionalData.Dimensions.LookupArrays.ReverseOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float32}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDsource, Dict{String, Any}}, EPSG, EPSG, Y{Colon}}}, Ti{DimensionalData.Dimensions.LookupArrays.Sampled{DateTime, Vector{DateTime}, DimensionalData.Dimensions.LookupArrays.Unordered, DimensionalData.Dimensions.LookupArrays.Irregular{Tuple{Nothing, Not…

It's not working with:

> ds_stack
Rasters.RasterStack{NamedTuple{(:ileadfra,), Tuple{Rasters.FileArray{Rasters.NCDsource, Union{Missing, Float32}, 3, Symbol, DiskArrays.GridChunks{3}, DiskArrays.Unchunked}}}, Tuple{}, Tuple{}, NamedTuple{(:ileadfra,), Tuple{Tuple{}}}, DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDsource, Dict{String, Any}}, NamedTuple{(:ileadfra,), Tuple{DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDsource, Dict{String, Any}}}}, NamedTuple{(:ileadfra,), Tuple{Float64}}}((ileadfra = Disk Array with size 1442 x 1021 x 1
,), (), (), (ileadfra = (),), DimensionalData.Dimensions.LookupArrays.Metadata{Rasters.NCDsource, Dict{String, Any}}(Dict{String, Any}("NCO" => "netCDF Operators version 4.9.2 (Homepage = http:https://nco.sf.net, Code = http:https://github.com/nco/nco)", "source" => "ORAS5 - backward extension (ECMWF)", "reference" => "https://www.ecmwf.int/en/research/climate-reanalysis/ocean-reanalysis", "title" => "Monthly mean sea ice fields", "institution" => "European Centre for Medi…

[rafaqz]

First, zoom out a bit. What is your actual goal? What is this for? I have no idea what you want to do here.

[ThomasAuriel]

I'm downloading data from copernicus:

• 2m air temperature (t2m) from here: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-era5-single-levels?tab=overview
• Sea Ice concentration (ileadfra) here: https://cds.climate.copernicus.eu/cdsapp#!/dataset/reanalysis-oras5?tab=form

I'm downloading data over a long period (a year) and want to split the file in subDs (one for each time). I do this to make data manipulation easier (I have many files and it is simpler to request files based on their filename which are defined using time and variable names).

The code above works well with t2m variable but not with ileadfra. This second variable comes with several variables in the same file (:ileadfra, :nav_lat, :nav_lon)

[rafaqz]

Ok. The code is not a real MWE so I cant really see what its doing.

If you want me to do anything with it you need to post a complete working MWE with downloads included in the code. Run it yourself from scratch in a new session to make sure it works and I can just paste it and it runs (or hits your error)

Also paste the full error you get here

[ThomasAuriel]

I'm not able to reproduce the issue in a shorter code.

using Random
using Rasters
import NCDatasets

# Assuming ./tmp/CDS_Product_Global_2mAirTemperature_[0]_20210101000000_20210102000000.nc exist
# Assuming ./tmp/ileadfra_control_monthly_highres_2D_200001_CONS_v0.1.nc exist
# They can be provided

# Assuming ./input exists

listFiles1 = [
    "./input/tmp/CDS_Product_Global_2mAirTemperature_[0]_20210101000000_20210102000000.nc",
]
listFiles2 = [
    "./input/tmp/ileadfra_control_monthly_highres_2D_200001_CONS_v0.1.nc"
]
variables1 = [:t2m]
variables2 = [:ileadfra]

dataFilepath = "./input/data"

# for (file, variable) in [(listFiles1, variables1), (listFiles2, variables2)]
for (file, variable) in [(listFiles2, variables2)]
    # Open file
    file = "./input/tmp/93a3e830/CDS_Product_Sea_IceConcentration_[0]_20000101000000_20000101000000/ileadfra_control_monthly_highres_2D_200001_CONS_v0.1.nc"
    ds_stack = RasterStack(file; name=variable, lazy=true, dropband=false, missingval=NaN, source=:netcdf)
    # Get all variable
    # variables = map(string, keys(ds_stack))

    ## Store data
    for v in variable

        try
            ds = ds_stack[v]
            # Define folder to store variable
            name = string(ds.name)
            print("Save $name")

            for time in lookup(ds, Ti)

                try
                    # Define filepath
                    filepath = joinpath(dataFilepath, "$(string(v))_0_$(Dates.format(time, "yyyymmddTHHMMSS")).nc")
                    println("Save file to $filepath")

                    ## Store element
                    subDs = ds[Ti=At([time])]
                
                    write(filepath, subDs, force=true)

                    println("done")
                catch ex
                    println("Error during processing: $(ex)")
                end

            end
        catch ex
            # Variable is not present
        end
    end
    
end

This shorter code works fine with both files.

What I don't understand is why the two outputs with the whole code (the two output in the first message) are differents. This is probably by my code but I don't see where it comes from. I don't understand what the different between the two.

[rafaqz]

Thanks. Are you just defining file a second time inside the loop or is that a typo?

You also dont use variables1?

If you need any more help I need to be able to run in.

Its always best if you download the files with download right in the MWE. If you want me to run it and find the problem it needs to be easy and completely reproducable.