How to pass kw args to create cog #535

alex-s-gardner · 2023-10-02T23:38:16Z

I'm struggling to figure out how to pass kw args to Rasters.write to create a cloud optimized geotiff file

This works as expected:

using Rasters
using GeoFormatTypes

epsg = GeoFormatTypes.EPSG("EPSG:32609")
gridsize = 250;
min_x = 273500;
max_x = 2507100;
min_y = 4077900;
max_y = 7234200;

x = X(min_x:gridsize:max_x);
y = Y(min_y:gridsize:max_y);
A = Raster(zeros(UInt8,y,x))
setcrs(A, epsg)

Rasters.write("out.tif", A)

But I can't figure out how to pass the relevant kw args to create a cog:

Rasters.write("out.tif", A; tiled = true,  compress="ZSTD")
ERROR: MethodError: no method matching write(::String, ::Matrix{Float64}; tiled::Bool, compress::String)

Closest candidates are:
  write(::AbstractString, ::Any, ::Any...) got unsupported keyword arguments "tiled", "compress"
   @ Base io.jl:459
  write(::AbstractString, ::AbstractRasterSeries; ext, source, kw...)
   @ Rasters ~/.julia/packages/Rasters/9ST4J/src/write.jl:76
  write(::AbstractString, ::AbstractRasterStack; suffix, ext, source, verbose, kw...)
   @ Rasters ~/.julia/packages/Rasters/9ST4J/src/write.jl:34
  ...

Stacktrace:
 [1] top-level scope
   @ REPL[84]:1

I'm finding it hard to trace where the kw are parsed and handed to the writer.

The text was updated successfully, but these errors were encountered:

rafaqz · 2023-10-03T08:15:31Z

setcrs does not mutate it rebuilds (no bang ! And we follow the rules here) so this does nothing: setcrs(A, epsg)

Just use crs keyword to Raster. This is all in the Raster docs.

To create a COG you need driver="COG" and you also may need Raster master because I recently fixed some bugs in it.

Please read the write docs for the rest, there are no tiled or compress keywords because they are only for GTiff.

Instead there is options which takes any GDAL options specific to each driver. The link to find them is in the write docs. Bu you need e.g. "BLOCKSIZE"=>256 for COG.

alex-s-gardner · 2023-10-03T23:24:38Z

Thanks for this. Looking through the docs all I see is the API Reference for .write and I don't see any information about the options kw or that it takes a Tuple of pairs. I also don't see anything about a driver kw and what options for drivers there are... maybe I'm looking in the wrong place?

rafaqz · 2023-10-04T03:08:30Z

I think its not printing the GDAL specific docs. Maybe they should all go in one doc.

But I meant ?write and you should see the docs for GDALsouce

options=Dict("BLOCKSIZE"=>256)

alex-s-gardner · 2023-10-04T03:12:21Z

Here's the Achilles heal of multiple dispatch at work:

help?> write
search: write closewrite unsafe_write iswritable showgradients lowercasefirst

  write(io::IO, x)
  write(filename::AbstractString, x)

  Write the canonical binary representation of a value to the given I/O stream or file. Return the number of bytes written into the stream. See also print to write a text representation (with an encoding that may depend upon io).

  The endianness of the written value depends on the endianness of the host system. Convert to/from a fixed endianness when writing/reading (e.g. using htol and ltoh) to get results that are consistent across platforms.

  You can write multiple values with the same write call. i.e. the following are equivalent:

  write(io, x, y...)
  write(io, x) + write(io, y...)

  Examples
  ≡≡≡≡≡≡≡≡≡≡

  Consistent serialization:

  julia> fname = tempname(); # random temporary filename
  
  julia> open(fname,"w") do f
             # Make sure we write 64bit integer in little-endian byte order
             write(f,htol(Int64(42)))
         end
  8
  
  julia> open(fname,"r") do f
             # Convert back to host byte order and host integer type
             Int(ltoh(read(f,Int64)))
         end
  42

  Merging write calls:

  julia> io = IOBuffer();
  
  julia> write(io, "JuliaLang is a GitHub organization.", " It has many members.")
  56
  
  julia> String(take!(io))
  "JuliaLang is a GitHub organization. It has many members."
  
  julia> write(io, "Sometimes those members") + write(io, " write documentation.")
  44
  
  julia> String(take!(io))
  "Sometimes those members write documentation."

  User-defined plain-data types without write methods can be written when wrapped in a Ref:

  julia> struct MyStruct; x::Float64; end
  
  julia> io = IOBuffer()
  IOBuffer(data=UInt8[...], readable=true, writable=true, seekable=true, append=false, size=0, maxsize=Inf, ptr=1, mark=-1)
  
  julia> write(io, Ref(MyStruct(42.0)))
  8
  
  julia> seekstart(io); read!(io, Ref(MyStruct(NaN)))
  Base.RefValue{MyStruct}(MyStruct(42.0))

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  write(::IO, ::Message)

  Write start line, headers and body of HTTP Message.

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  Base.write(fp::S3Path, content::String; kwargs...)
  Base.write(fp::S3Path, content::Vector{UInt8}; part_size_mb=50, multipart::Bool=true,
             returns::Symbol=:parsed, other_kwargs...,)

  Write content to S3Path fp.

  Optional Arguments
  ≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡≡

    •  multipart: when true, uploads data via s3_multipart_upload for content greater than part_size_mb bytes; when false, or when content is shorter than part_size_mb, uploads data via s3_put.

    •  part_size_mb: when multipart=true, sets maximum length of partitioned data (in bytes).

    •  returns: determines the result returned by the function: :response (the AWS API response), :parsed(default; the parsed AWS API response), or:path(the newly created [S3Path](@ref), including itsversion` when versioning is enabled for the bucket).

    •  other_kwargs: additional kwargs passed through into s3_multipart_upload.

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  Base.write(filename::AbstractString, A::AbstractRaster; kw...)

  Write an AbstractRaster to file, guessing the backend from the file extension.

  Keyword arguments are passed to the write method for the backend.

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  Base.write(filename::AbstractString, s::AbstractRasterStack; suffix, kw...)

  Write any AbstractRasterStack to file, guessing the backend from the file extension.

  Keywords
  ==========

    •  suffix: suffix to append to file names. By default the layer key is used.

  Other keyword arguments are passed to the write method for the backend.

  If the source can't be saved as a stack-like object, individual array layers will be saved.

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  Base.write(filepath::AbstractString, s::AbstractRasterSeries; kw...)

  Write any AbstractRasterSeries to file, guessing the backend from the file extension.

  The lookup values of the series will be appended to the filepath (before the extension), separated by underscores.

  Keywords
  ==========

  See other docs for write. All keywords are passed through to Raster and RasterStack methods.

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  Base.write(filename::AbstractString, ::Type{GRDsource}, s::AbstractRaster; force=false)

  Write a Raster to a .grd file with a .gri header file. The extension of filename will be ignored.

  Returns filename.

  ────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

  Base.write(filename::AbstractString, ::Type{GDALsource}, A::AbstractRaster; force=false, kw...)

  Write a Raster to file using GDAL.

  Keywords
  ≡≡≡≡≡≡≡≡≡≡

    •  driver: A GDAL driver name or a GDAL driver retrieved via ArchGDAL.getdriver(drivername). Guessed from the filename extension by default.

    •  options::Dict{String,String}: A dictionary containing the dataset creation options passed to the driver. For example: Dict("COMPRESS"=>"DEFLATE")
       Valid options for the drivers can be looked up here: https://gdal.org/drivers/raster/index.html

  Returns filename.

alex-s-gardner · 2023-10-04T03:16:46Z

But it is indeed there at the end, but it's not easy for a user to figure out and find. My favorite is when documentation contains examples. I would think that many users won't even know what GDAL is. Now I know how I can contibute if I every can squeeze out some more time... maybe a get place for an intern to contibute.

alex-s-gardner · 2023-10-04T03:17:55Z

Also, this works:

Rasters.write(dhdt_file * ".tif", A; driver="COG", force=true)
┌ Warning: `missing` cant be written to .tif, missinval for `Float64` of `-Inf` used instead
└ @ Rasters ~/.julia/packages/Rasters/9ST4J/src/utils.jl:32
"/mnt/bylot-r3/data/height_change/WNA/2000_2022/lat[+50+52]lon[-128-126].cop30_v2.tif"

but not this:

Rasters.write(dhdt_file * ".tif", A; driver="COG", options=Dict("BLOCKSIZE" => 256), force=true)
┌ Warning: `missing` cant be written to .tif, missinval for `Float64` of `-Inf` used instead
└ @ Rasters ~/.julia/packages/Rasters/9ST4J/src/utils.jl:32
ERROR: MethodError: Cannot `convert` an object of type String to an object of type Int64

Closest candidates are:
  convert(::Type{T}, ::ColorTypes.Gray24) where T<:Real
   @ ColorTypes ~/.julia/packages/ColorTypes/1dGw6/src/conversions.jl:114
  convert(::Type{T}, ::ColorTypes.Gray) where T<:Real
   @ ColorTypes ~/.julia/packages/ColorTypes/1dGw6/src/conversions.jl:113
  convert(::Type{T}, ::Ptr) where T<:Integer
   @ Base pointer.jl:23
  ...

Stacktrace:
  [1] setindex!(h::Dict{String, Int64}, v0::String, key::String)
    @ Base ./dict.jl:369
  [2] _gdal_process_options(driver::String, options::Dict{String, Int64}; _block_template::Raster{Float64, 2, Tuple{X{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, X{Colon}}}, Y{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ReverseOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, Y{Colon}}}}, Tuple{}, Matrix{Float64}, DimensionalData.NoName, DimensionalData.Dimensions.LookupArrays.NoMetadata, Float64})
    @ RastersArchGDALExt ~/.julia/packages/Rasters/9ST4J/ext/RastersArchGDALExt/gdal_source.jl:435
  [3] _gdal_process_options
    @ ~/.julia/packages/Rasters/9ST4J/ext/RastersArchGDALExt/gdal_source.jl:429 [inlined]
  [4] _gdal_with_driver(f::RastersArchGDALExt.var"#39#41"{Raster{Float64, 2, Tuple{X{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, X{Colon}}}, Y{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ReverseOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, Y{Colon}}}}, Tuple{}, Matrix{Float64}, DimensionalData.NoName, DimensionalData.Dimensions.LookupArrays.NoMetadata, Float64}}, filename::String, driver::String, create_kw::NamedTuple{(:width, :height, :nbands, :dtype), Tuple{Int64, Int64, Int64, DataType}}; options::Dict{String, Int64}, _block_template::Raster{Float64, 2, Tuple{X{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, X{Colon}}}, Y{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ReverseOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, Y{Colon}}}}, Tuple{}, Matrix{Float64}, DimensionalData.NoName, DimensionalData.Dimensions.LookupArrays.NoMetadata, Float64})
    @ RastersArchGDALExt ~/.julia/packages/Rasters/9ST4J/ext/RastersArchGDALExt/gdal_source.jl:406
  [5] _gdal_with_driver
    @ ~/.julia/packages/Rasters/9ST4J/ext/RastersArchGDALExt/gdal_source.jl:403 [inlined]
  [6] #_gdalwrite#38
    @ ~/.julia/packages/Rasters/9ST4J/ext/RastersArchGDALExt/gdal_source.jl:391 [inlined]
  [7] _gdalwrite
    @ ~/.julia/packages/Rasters/9ST4J/ext/RastersArchGDALExt/gdal_source.jl:386 [inlined]
  [8] write(filename::String, ::Type{Rasters.GDALsource}, A::Raster{Float64, 2, Tuple{X{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, X{Colon}}}, Y{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, Y{Colon}}}}, Tuple{}, Matrix{Float64}, DimensionalData.NoName, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; force::Bool, verbose::Bool, kw::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol}, NamedTuple{(:driver, :options), Tuple{String, Dict{String, Int64}}}})
    @ RastersArchGDALExt ~/.julia/packages/Rasters/9ST4J/ext/RastersArchGDALExt/gdal_source.jl:73
  [9] write
    @ ~/.julia/packages/Rasters/9ST4J/ext/RastersArchGDALExt/gdal_source.jl:64 [inlined]
 [10] write(filename::String, A::Raster{Float64, 2, Tuple{X{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, X{Colon}}}, Y{Projected{Float64, StepRangeLen{Float64, Base.TwicePrecision{Float64}, Base.TwicePrecision{Float64}, Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Float64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata, String, Nothing, Y{Colon}}}}, Tuple{}, Matrix{Float64}, DimensionalData.NoName, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; source::Type, kw::Base.Pairs{Symbol, Any, Tuple{Symbol, Symbol, Symbol}, NamedTuple{(:driver, :options, :force), Tuple{String, Dict{String, Int64}, Bool}}})
    @ Rasters ~/.julia/packages/Rasters/9ST4J/src/write.jl:11
 [11] top-level scope
    @ REPL[39]:1

rafaqz · 2023-10-04T03:21:16Z

Oh its "256".

Probably we should auto convert that before sending to GDAL

rafaqz · 2023-10-04T03:26:07Z

My favorite is when documentation contains examples.

The problem is when would I write that... Its usually a choice between docs or bugfixes. I dont even have Rasters updated for the new DD at present.

But you are most welcome to go through and reoranganise write into one docstring and add some examples!

alex-s-gardner · 2023-10-04T03:40:22Z

@rafaqz thanks for all the help.. and the library... very much appreciated. Docs will get there with time.. hopefully I can help with that

alex-s-gardner · 2023-10-04T22:41:08Z

@rafaqz is

Base.write(filename::AbstractString, ::Type{GDALsource}, A::AbstractRaster; force=false, kw...)
  Write a Raster to file using GDAL.

owned by ArchGDAL? I can't find it in Rasters

rafaqz · 2023-10-05T05:22:38Z

Its in the extension RastersArchGDALExt. Theres one for NCDatasets too in its extension, and grd but thats in the sources folder

maxfreu · 2023-10-22T11:18:13Z

Oh its "256".

Probably we should auto convert that before sending to GDAL

That's not the problem; the problem was setting the default value of "COMPRESS" to "ZSTD" in a Dict{String, Int}.

maxfreu mentioned this issue Oct 22, 2023

auto-convert option dicts to Dict{String, String} #542

Merged

rafaqz closed this as completed Oct 23, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

How to pass kw args to create cog #535

How to pass kw args to create cog #535

alex-s-gardner commented Oct 2, 2023 •

edited

Loading

rafaqz commented Oct 3, 2023 •

edited

Loading

alex-s-gardner commented Oct 3, 2023

rafaqz commented Oct 4, 2023 •

edited

Loading

alex-s-gardner commented Oct 4, 2023

alex-s-gardner commented Oct 4, 2023

alex-s-gardner commented Oct 4, 2023 •

edited

Loading

rafaqz commented Oct 4, 2023 •

edited

Loading

rafaqz commented Oct 4, 2023 •

edited

Loading

alex-s-gardner commented Oct 4, 2023

alex-s-gardner commented Oct 4, 2023

rafaqz commented Oct 5, 2023 •

edited

Loading

maxfreu commented Oct 22, 2023

How to pass kw args to create cog #535

How to pass kw args to create cog #535

Comments

alex-s-gardner commented Oct 2, 2023 • edited Loading

rafaqz commented Oct 3, 2023 • edited Loading

alex-s-gardner commented Oct 3, 2023

rafaqz commented Oct 4, 2023 • edited Loading

alex-s-gardner commented Oct 4, 2023

alex-s-gardner commented Oct 4, 2023

alex-s-gardner commented Oct 4, 2023 • edited Loading

rafaqz commented Oct 4, 2023 • edited Loading

rafaqz commented Oct 4, 2023 • edited Loading

alex-s-gardner commented Oct 4, 2023

alex-s-gardner commented Oct 4, 2023

rafaqz commented Oct 5, 2023 • edited Loading

maxfreu commented Oct 22, 2023

alex-s-gardner commented Oct 2, 2023 •

edited

Loading

rafaqz commented Oct 3, 2023 •

edited

Loading

rafaqz commented Oct 4, 2023 •

edited

Loading

alex-s-gardner commented Oct 4, 2023 •

edited

Loading

rafaqz commented Oct 4, 2023 •

edited

Loading

rafaqz commented Oct 4, 2023 •

edited

Loading

rafaqz commented Oct 5, 2023 •

edited

Loading