Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

optimise extract performance #641

Merged
merged 4 commits into from
Apr 30, 2024
Merged

optimise extract performance #641

merged 4 commits into from
Apr 30, 2024

Conversation

rafaqz
Copy link
Owner

@rafaqz rafaqz commented Apr 24, 2024

Closes #625

@alex-s-gardner this PR should fix some of your performance concerns, looks like at least an order of magnitude faster, a million points should take a fraction of a second.

@rafaqz rafaqz force-pushed the extract branch 2 times, most recently from 0f17a1f to e6ebbef Compare April 29, 2024 17:27
@alex-s-gardner
Copy link
Contributor

alex-s-gardner commented Apr 29, 2024

extract no longer works if pts = Vector{Tuple{Float64, Float64}}

using GeoInterface
using Rasters

lon, lat = X(25:.1:30), Y(25:.1:30);
mask0 = Raster(falses(lon, lat));

lon = rand(25:0.1:30, 100);
lat = rand(25:0.1:30, 100);

pts1 = GeoInterface.PointTuple.([((Y=y, X=x)) for (x, y) in
    zip(lon, lat)]);

pts2 = extract(mask0, pts1, atol=0.0001, index=true, geometry=false);
ERROR: MethodError: Cannot `convert` an object of type X{Contains{Float64}} to an object of type Int64

Closest candidates are:
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:84
  convert(::Type{T}, ::LoopVectorization.UpperBoundedInteger) where T<:Integer
   @ LoopVectorization ~/.julia/packages/LoopVectorization/1hqld/src/reconstruct_loopset.jl:30
  convert(::Type{T}, ::EzXML.NodeType) where T<:Integer
   @ EzXML ~/.julia/packages/EzXML/DL8na/src/node.jl:36
  ...

Stacktrace:
  [1] cvt1
    @ ./essentials.jl:468 [inlined]
  [2] ntuple
    @ ./ntuple.jl:49 [inlined]
  [3] convert(::Type{Tuple{Int64, Int64}}, x::Tuple{X{Contains{Float64}}, Y{Contains{Float64}}})
    @ Base ./essentials.jl:470
  [4] convert(::Type{Union{Missing, Tuple{Int64, Int64}}}, x::Tuple{X{Contains{Float64}}, Y{Contains{Float64}}})
    @ Base ./missing.jl:70
  [5] cvt1
    @ ./essentials.jl:468 [inlined]
  [6] ntuple(f::Base.var"#cvt1#1"{Tuple{Union{}, Union{}}, Tuple{Tuple{}, UInt8}}, ::Val{2})
    @ Base ./ntuple.jl:49
  [7] convert
    @ ./essentials.jl:470 [inlined]
  [8] Tuple
    @ ./namedtuple.jl:201 [inlined]
  [9] convert
    @ ./namedtuple.jl:196 [inlined]
 [10] _maybe_add_fields
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:232 [inlined]
 [11] #_extract_point#734
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:221 [inlined]
 [12] _extract_point
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:191 [inlined]
 [13] _extract(A::Raster{…}, ::Nothing, geoms::Vector{…}; names::@NamedTuple{}, skipmissing::DimensionalData.Dimensions.Lookups._False, kw::@Kwargs{})
    @ Rasters ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:117
 [14] _extract
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:98 [inlined]
 [15] _extract
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:94 [inlined]
 [16] #_extract#693
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:86 [inlined]
 [17] #extract#692
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:66 [inlined]
 [18] top-level scope
    @ REPL[23]:1
Some type information was truncated. Use `show(err)` to see complete types.

Same error when not using GeoInterface:

pts1 = ([((y, x)) for (x, y) in zip(lon, lat)]);
pts2 = extract(mask0, pts1, atol=0.0001, index=true, geometry=false);
ERROR: MethodError: Cannot `convert` an object of type X{At{Float64, Float64, Nothing}} to an object of type Int64

Closest candidates are:
  convert(::Type{T}, ::T) where T
   @ Base Base.jl:84
  convert(::Type{T}, ::LoopVectorization.UpperBoundedInteger) where T<:Integer
   @ LoopVectorization ~/.julia/packages/LoopVectorization/1hqld/src/reconstruct_loopset.jl:30
  convert(::Type{T}, ::EzXML.NodeType) where T<:Integer
   @ EzXML ~/.julia/packages/EzXML/DL8na/src/node.jl:36
  ...

Stacktrace:
  [1] cvt1
    @ ./essentials.jl:468 [inlined]
  [2] ntuple
    @ ./ntuple.jl:49 [inlined]
  [3] convert(::Type{Tuple{Int64, Int64}}, x::Tuple{X{At{Float64, Float64, Nothing}}, Y{At{Float64, Float64, Nothing}}})
    @ Base ./essentials.jl:470
  [4] convert(::Type{Union{Missing, Tuple{…}}}, x::Tuple{X{At{…}}, Y{At{…}}})
    @ Base ./missing.jl:70
  [5] cvt1
    @ ./essentials.jl:468 [inlined]
  [6] ntuple(f::Base.var"#cvt1#1"{Tuple{Union{…}, Union{…}}, Tuple{Tuple{…}, Bool}}, ::Val{2})
    @ Base ./ntuple.jl:49
  [7] convert
    @ ./essentials.jl:470 [inlined]
  [8] Tuple
    @ ./namedtuple.jl:201 [inlined]
  [9] convert
    @ ./namedtuple.jl:196 [inlined]
 [10] _maybe_add_fields
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:232 [inlined]
 [11] _extract_point(::Type{…}, x::Raster{…}, point::Tuple{…}; dims::Tuple{…}, names::@NamedTuple{…}, atol::Float64, kw::@Kwargs{…})
    @ Rasters ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:221
 [12] _extract_point
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:191 [inlined]
 [13] _extract(A::Raster{…}, ::Nothing, geoms::Vector{…}; names::@NamedTuple{…}, skipmissing::DimensionalData.Dimensions.Lookups._False, kw::@Kwargs{…})
    @ Rasters ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:117
 [14] _extract
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:98 [inlined]
 [15] _extract
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:94 [inlined]
 [16] #_extract#693
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:86 [inlined]
 [17] #extract#692
    @ ~/.julia/packages/Rasters/deZXg/src/methods/extract.jl:66 [inlined]
 [18] top-level scope
    @ REPL[66]:1
Some type information was truncated. Use `show(err)` to see complete types.

@rafaqz
Copy link
Owner Author

rafaqz commented Apr 29, 2024

In progress PR still failing tests ?

Copy link

codecov bot commented Apr 30, 2024

Codecov Report

Attention: Patch coverage is 82.03593% with 30 lines in your changes are missing coverage. Please review.

Project coverage is 82.48%. Comparing base (a15ebb1) to head (dc7ac0e).
Report is 3 commits behind head on main.

Files Patch % Lines
src/methods/extract.jl 82.03% 30 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##             main     #641      +/-   ##
==========================================
+ Coverage   82.32%   82.48%   +0.15%     
==========================================
  Files          60       60              
  Lines        4357     4459     +102     
==========================================
+ Hits         3587     3678      +91     
- Misses        770      781      +11     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

@rafaqz rafaqz merged commit a3e139f into main Apr 30, 2024
2 of 5 checks passed
@rafaqz rafaqz deleted the extract branch April 30, 2024 22:37
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Optimise extract for Regular lookups
2 participants