Plotting RGB Rasters with Makie

[tiemvanderdeure]

I used to be able to plot RGBA rasters with Makie using heatmap, but some recent version apparently broke this. I now get an error message that says

Makie needs to convert all plot input arguments to types that can be consumed by the backends (typically Arrays with Float32 elements).
You can define a method for `Makie.convert_arguments` (a type recipe) for these types or their supertypes to make this set of arguments convertible (See http:https://docs.makie.org/stable/documentation/recipes/index.html).

Converting to an Array and then plotting still works. Any idea what broke this?

This triggers the error:

using Rasters, GLMakie, Colors
xdim = X(1:10); ydim = Y(1:10)
rgba_raster = Raster([RGBA(i,j,0,1) for i in 0.1:0.1:1, j in 0.1:0.1:1], (xdim, ydim))
heatmap(rgba_raster, color_map = rgba_raster)

[rafaqz]

Full stack trace always helps with those ideas ;)

[tiemvanderdeure]

Here you go!

Stacktrace:
  [1] _broadcast_getindex_evalf
    @ .\broadcast.jl:683 [inlined]
  [2] _broadcast_getindex
    @ .\broadcast.jl:656 [inlined]
  [3] getindex
    @ .\broadcast.jl:610 [inlined]
  [4] copy
    @ .\broadcast.jl:912 [inlined]
  [5] copy(bc::Base.Broadcast.Broadcasted{DimensionalData.DimensionalStyle{Base.Broadcast.DefaultArrayStyle{2}}, Tuple{DimensionalData.Dimensions.DimUnitRange{Int64, Base.OneTo{Int64}, X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, DimensionalData.Dimensions.DimUnitRange{Int64, Base.OneTo{Int64}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}}, typeof(Rasters._missing_or_float32), Tuple{Raster{Any, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{Any}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Float32}}})
    @ DimensionalData C:\Users\tsh371\.julia\packages\DimensionalData\4TpBG\src\array\broadcast.jl:39
  [6] materialize
    @ .\broadcast.jl:873 [inlined]
  [7] _prepare_makie(A::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing})
    @ Rasters C:\Users\tsh371\.julia\dev\Rasters\src\plotrecipes.jl:384
  [8] convert_arguments(#unused#::DiscreteSurface, raster::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing})
    @ Rasters C:\Users\tsh371\.julia\dev\Rasters\src\plotrecipes.jl:305
  [9] convert_arguments(T::Type{Heatmap{Tuple{Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}}}}, args::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; kw::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Makie C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\conversions.jl:10
 [10] convert_arguments
    @ C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\conversions.jl:7 [inlined]
 [11] plot!(scene::Scene, P::Type{Heatmap}, attributes::Attributes, args::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; kw_attributes::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
    @ Makie C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\interfaces.jl:292
 [12] plot!
    @ C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\interfaces.jl:275 [inlined]
 [13] plot(P::Type{Heatmap}, args::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; axis::NamedTuple{(), Tuple{}}, figure::NamedTuple{(), Tuple{}}, kw_attributes::Base.Pairs{Symbol, Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, 
DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}, Tuple{Symbol}, NamedTuple{(:color_map,), Tuple{Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}}}})
    @ Makie C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\figureplotting.jl:48
 [14] plot
    @ C:\Users\tsh371\.julia\packages\Makie\Ylh0P\src\figureplotting.jl:31 [inlined]
 [15] #heatmap#18
    @ C:\Users\tsh371\.julia\packages\MakieCore\zgrOY\src\recipes.jl:34 [inlined]
 [16] top-level scope

[rafaqz]

Yeah seems we are not handling colors... this will all move to DimensionalData.jl very soon though, we can fix there

[tiemvanderdeure]

All right. I figured out just passing dimensions as x and y arguments fixes it, even though I don't really understand why.

heatmap(lookup(dims(rgba_raster, (X, Y)))..., rgba_raster) works.

[rafaqz]

Yeah its just routing around the explicit method that does that.

Although do realise that can put the pixels in the wrong place... gdal lookup values are at the start of the pixel not the middle.

[felixcremer]

This fails currently, because we explicitly convert the data to Float32 in

Rasters.jl/ext/RastersMakieExt/plotrecipes.jl

Line 350 in 7eec50c

isequal(x, missingval(A)) || ismissing(x) ? NaN32 : Float32(x)

.

If I remove the Float32 there, image(rgba_raster) works and the heatmap call fails while constructing the Colorbar see error below.
Maybe we could special case the _prepare dim array function for RGB elementtypes.

julia> heatmap(rgba_raster, colormap = rgba_raster)
ERROR: Neither heatmap nor any of its children use a colormap. Cannot create a Colorbar from this plot, please create it manually.
        If this is a recipe, one needs to overload `Makie.extract_colormap(::Heatmap)` to allow for the automatical creation of a Colorbar.
Stacktrace:
 [1] error(s::String)
   @ Base ./error.jl:35
 [2] Colorbar(fig_or_scene::GridPosition, plot::Heatmap{Tuple{Vector{Float32}, Vector{Float32}, Matrix{RGBA{Float32}}}}; kwargs::Base.Pairs{Symbol, String, Tuple{Symbol}, NamedTuple{(:label,), Tuple{String}}})
   @ Makie ~/.julia/packages/Makie/NrrbP/src/makielayout/blocks/colorbar.jl:93
 [3] heatmap(A::Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}; x::Nothing, y::Nothing, colorbarkw::NamedTuple{(), Tuple{}}, attributes::Base.Pairs{Symbol, Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}, Tuple{Symbol}, NamedTuple{(:colormap,), Tuple{Raster{RGBA{Float64}, 2, Tuple{X{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}, Y{DimensionalData.Dimensions.LookupArrays.Sampled{Int64, UnitRange{Int64}, DimensionalData.Dimensions.LookupArrays.ForwardOrdered, DimensionalData.Dimensions.LookupArrays.Regular{Int64}, DimensionalData.Dimensions.LookupArrays.Points, DimensionalData.Dimensions.LookupArrays.NoMetadata}}}, Tuple{}, Matrix{RGBA{Float64}}, Symbol, DimensionalData.Dimensions.LookupArrays.NoMetadata, Missing}}}})
   @ DimensionalDataMakie ~/Documents/DDPlotting/dev/DimensionalData/ext/DimensionalDataMakie.jl:147
 [4] top-level scope
   @ REPL[10]:1

[rafaqz]

Hmm what does Makie do for missing value in RGB? transparency?

We could pass missing but then makie will have to transform that again.

I guess the current method should dispatch on Real, and we can have a fallback that just uses missing