Type piracy? #8

giordano · 2020-11-20T01:05:30Z

It looks like there is type piracy somewhere related to this package:

(@v1.6) pkg> activate --temp
  Activating new environment at `/tmp/jl_FXOiIk/Project.toml`

(jl_FXOiIk) pkg> st
Status `/tmp/jl_FXOiIk/Project.toml` (empty project)

(jl_FXOiIk) pkg> add HCIDatasets
    Updating registry at `~/.julia/registries/General`
    Updating git-repo `https://github.com/JuliaRegistries/General.git`
   Resolving package versions...
Updating `/tmp/jl_FXOiIk/Project.toml`
  [1893dc6a] + HCIDatasets v0.2.0
Updating `/tmp/jl_FXOiIk/Manifest.toml`
  [3b1b4be9] + CFITSIO v1.0.0
  [124859b0] + DataDeps v0.7.3
  [525bcba6] + FITSIO v0.16.3
  [1893dc6a] + HCIDatasets v0.2.0
  [cd3eb016] + HTTP v0.8.19
  [83e8ac13] + IniFile v0.5.0
  [692b3bcd] + JLLWrappers v1.1.3
  [739be429] + MbedTLS v1.0.3
  [189a3867] + Reexport v0.2.0
  [b3e40c51] + CFITSIO_jll v3.48.0+0
  [c8ffd9c3] + MbedTLS_jll v2.24.0+1
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll
  [14a3606d] + MozillaCACerts_jll

julia> using HCIDatasets

(jl_FXOiIk) pkg> st
ERROR: MethodError: no method matching push!!(::Tuple{}, ::String)
Closest candidates are:
  push!!(::Vector{T} where T, ::Any) at toml_parser.jl:659
Stacktrace:
  [1] parse_array(l::Base.TOML.Parser)
    @ Base.TOML ./toml_parser.jl:683
  [2] parse_value(l::Base.TOML.Parser)
    @ Base.TOML ./toml_parser.jl:634
  [3] parse_entry(l::Base.TOML.Parser, d::Dict{String, Any})
    @ Base.TOML ./toml_parser.jl:157
  [4] parse_toplevel(l::Base.TOML.Parser)
    @ Base.TOML ./toml_parser.jl:157
  [5] tryparse(l::Base.TOML.Parser)
    @ Base.TOML ./toml_parser.jl:449
  [6] tryparsefile(f::String)
    @ TOML ~/repo/julia/usr/share/julia/stdlib/v1.6/TOML/src/TOML.jl:57
  [7] read_manifest(f_or_io::String)
    @ Pkg.Types ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/manifest.jl:165
  [8] Pkg.Types.EnvCache(env::Nothing)
    @ Pkg.Types ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:280
  [9] EnvCache
    @ ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/Types.jl:260 [inlined]
 [10] Pkg.Types.Context()
    @ Pkg.Types ./util.jl:444
 [11] #status#78
    @ ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:72 [inlined]
 [12] status(pkgs::Vector{Pkg.Types.PackageSpec})
    @ Pkg.API ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/API.jl:72
 [13] do_cmd!(command::Pkg.REPLMode.Command, repl::REPL.LineEditREPL)
    @ Pkg.REPLMode ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:408
 [14] do_cmd(repl::REPL.LineEditREPL, input::String; do_rethrow::Bool)
    @ Pkg.REPLMode ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:386
 [15] do_cmd
    @ ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:377 [inlined]
 [16] (::Pkg.REPLMode.var"#24#27"{REPL.LineEditREPL, REPL.LineEdit.Prompt})(s::REPL.LineEdit.MIState, buf::IOBuffer, ok::Bool)
    @ Pkg.REPLMode ~/repo/julia/usr/share/julia/stdlib/v1.6/Pkg/src/REPLMode/REPLMode.jl:550
 [17] #invokelatest#2
    @ ./essentials.jl:707 [inlined]
 [18] invokelatest
    @ ./essentials.jl:706 [inlined]
 [19] run_interface(terminal::REPL.Terminals.TextTerminal, m::REPL.LineEdit.ModalInterface, s::REPL.LineEdit.MIState)
    @ REPL.LineEdit ~/repo/julia/usr/share/julia/stdlib/v1.6/REPL/src/LineEdit.jl:2441
 [20] run_frontend(repl::REPL.LineEditREPL, backend::REPL.REPLBackendRef)
    @ REPL ~/repo/julia/usr/share/julia/stdlib/v1.6/REPL/src/REPL.jl:1125
 [21] (::REPL.var"#44#49"{REPL.LineEditREPL, REPL.REPLBackendRef})()
    @ REPL ./task.jl:395

This doesn't happen with FITSIO nor DataDeps only, which are the two dependencies of this package. Unless there is a bad interaction between them, I suspect the type piracy is in this package.

The text was updated successfully, but these errors were encountered:

giordano · 2020-11-20T01:07:15Z

It also happens only in Julia v1.6, but not v1.5

mileslucas · 2020-11-20T01:14:02Z

Uh oh, that's not good.

To start, I do a kind of nasty trick to lazy-load the data, and that's overloading Base.keys and Base.getindex for a type, like so-

abstract type Dataset end
struct BetaPictoris <: Dataset end

Base.getindex(::Type{BetaPictoris}, key) = # ... parse key and load data from file

the other possibility might be this method, which reaches into Datadeps.HTTP. This might be causing it because this is plugged into the DataDeps source which is evaluated on the module init.

HCIDatasets.jl/src/BetaPictoris.jl

Lines 20 to 27 in f7bf4f0

 function rename_download(remote_path, localdir) 

 update_period = DataDeps.progress_update_period() 

 DataDeps.HTTP.download( 

 remote_path, 

 joinpath(localdir, "naco_betapic_cube_empty.fits"); 

 update_period=update_period 

 ) 

 end

any ideas?

giordano · 2020-11-20T01:18:38Z

any ideas?

Binary search in the code of HCIDatasets would be my strategy to track down the culprit

giordano · 2020-11-20T01:21:16Z

I skimmed through the code, couldn't see anything blatantly pirat-y

KristofferC · 2020-11-20T09:31:39Z

HCIDatasets.jl/src/HCIDatasets.jl

Lines 14 to 15 in d52ceed

 Base.getindex(d::Type{<:Dataset}, keys...) = map(k -> d[k], keys) 

 Base.getindex(d::Type{<:Dataset}, keys) = map(k -> d[k], keys)

looks piraty to me since it also applies to Union.

HCIDatasets.jl/src/HCIDatasets.jl

Line 16 in d52ceed

Base.pairs(d::Type{<:Dataset}) = (k => d[k] for k in keys(d))

as well. It's not a good idea to define methods like this on types, better to use instances.

mileslucas · 2020-11-20T13:59:26Z

@KristofferC thanks for the tip. Do you think the piracy is removed if I avoid this-

Base.getindex(::Type{<:Dataset}, # ...

and instead write this?

Base.getindex(::Type{BetaPictoris}, # ...

KristofferC · 2020-11-20T14:10:35Z

I think it would be best to have

Base.getindex(::Dataset, ...)

and use singletons for the instances (like Nothing and nothing). But yes, using the concrete types like you wrote would be an improvement.

mileslucas · 2020-11-24T04:54:33Z

as of da2c812 I switched to using singletons, which worked just like I needed it to. I've since bumped to 0.2.1 and started a release. I'll try recreating the previous bug and seeing if this fixes it.

mileslucas · 2020-11-24T05:18:31Z

I am not able to reproduce the bug on my build of julia master @giordano

Julia Version 1.6.0-DEV.1567
Commit bdcaee033e* (2020-11-24 03:27 UTC)

using the exact same command order

pkg> activate --temp
pkg> st
pkg> add HCIDatasets
julia> using HCIDatasets
pkg> st

output (old version)

(@v1.6) pkg> activate --temp
  Activating new environment at `/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_J9SAHb/Project.toml`

(jl_J9SAHb) pkg> st
Status `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_J9SAHb/Project.toml` (empty project)

(jl_J9SAHb) pkg> add HCIDatasets
    Updating registry at `~/.julia/registries/General`
   Resolving package versions...
Updating `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_J9SAHb/Project.toml`
  [1893dc6a] + HCIDatasets v0.2.0
Updating `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_J9SAHb/Manifest.toml`
  [3b1b4be9] + CFITSIO v1.0.0
  [124859b0] + DataDeps v0.7.4
  [525bcba6] + FITSIO v0.16.3
  [1893dc6a] + HCIDatasets v0.2.0
  [cd3eb016] + HTTP v0.9.0
  [83e8ac13] + IniFile v0.5.0
  [692b3bcd] + JLLWrappers v1.1.3
  [739be429] + MbedTLS v1.0.3
  [189a3867] + Reexport v0.2.0
  [5c2747f8] + URIs v1.0.1
  [b3e40c51] + CFITSIO_jll v3.48.0+0
  [c8ffd9c3] + MbedTLS_jll v2.24.0+1
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll
  [14a3606d] + MozillaCACerts_jll

julia> using HCIDatasets

(jl_J9SAHb) pkg> st
Status `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_J9SAHb/Project.toml`
  [1893dc6a] HCIDatasets v0.2.0

output (new version)

(@v1.6) pkg> activate --temp
  Activating new environment at `/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_ssd5FJ/Project.toml`

(jl_ssd5FJ) pkg> st
Status `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_ssd5FJ/Project.toml` (empty project)

(jl_ssd5FJ) pkg> add HCIDatasets
    Updating registry at `~/.julia/registries/General`
   Resolving package versions...
   Installed HCIDatasets ─ v0.2.1
Updating `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_ssd5FJ/Project.toml`
  [1893dc6a] + HCIDatasets v0.2.1
Updating `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_ssd5FJ/Manifest.toml`
  [3b1b4be9] + CFITSIO v1.0.0
  [124859b0] + DataDeps v0.7.4
  [525bcba6] + FITSIO v0.16.3
  [1893dc6a] + HCIDatasets v0.2.1
  [cd3eb016] + HTTP v0.9.0
  [83e8ac13] + IniFile v0.5.0
  [692b3bcd] + JLLWrappers v1.1.3
  [739be429] + MbedTLS v1.0.3
  [189a3867] + Reexport v0.2.0
  [5c2747f8] + URIs v1.0.1
  [b3e40c51] + CFITSIO_jll v3.48.0+0
  [c8ffd9c3] + MbedTLS_jll v2.24.0+1
  [0dad84c5] + ArgTools
  [56f22d72] + Artifacts
  [2a0f44e3] + Base64
  [ade2ca70] + Dates
  [f43a241f] + Downloads
  [b77e0a4c] + InteractiveUtils
  [b27032c2] + LibCURL
  [76f85450] + LibGit2
  [8f399da3] + Libdl
  [56ddb016] + Logging
  [d6f4376e] + Markdown
  [ca575930] + NetworkOptions
  [44cfe95a] + Pkg
  [de0858da] + Printf
  [3fa0cd96] + REPL
  [9a3f8284] + Random
  [ea8e919c] + SHA
  [9e88b42a] + Serialization
  [6462fe0b] + Sockets
  [fa267f1f] + TOML
  [a4e569a6] + Tar
  [8dfed614] + Test
  [cf7118a7] + UUIDs
  [4ec0a83e] + Unicode
  [deac9b47] + LibCURL_jll
  [14a3606d] + MozillaCACerts_jll
Precompiling project...
  Progress [========================================>]  1/1
1 dependency successfully precompiled in 2 seconds (11 already precompiled)

julia> using HCIDatasets

(jl_ssd5FJ) pkg> st
Status `/private/var/folders/g2/n_lg14f56k776c2p35gktr000000gn/T/jl_ssd5FJ/Project.toml`
  [1893dc6a] HCIDatasets v0.2.1

KristofferC · 2020-11-24T09:49:59Z

I made the code in Julia less susceptible to this issue: JuliaLang/julia#38511.

giordano · 2020-11-24T09:54:51Z

I can confirm this is fixed now, thanks everybody!

KristofferC · 2020-11-24T09:56:14Z

It is not fixed though. It still type pirates it and any code using the same syntax I did to create such an array will get the same error here.

giordano closed this as completed Nov 24, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Type piracy? #8

Type piracy? #8

giordano commented Nov 20, 2020

giordano commented Nov 20, 2020

mileslucas commented Nov 20, 2020 •

edited

Loading

giordano commented Nov 20, 2020

giordano commented Nov 20, 2020

KristofferC commented Nov 20, 2020 •

edited

Loading

mileslucas commented Nov 20, 2020 •

edited

Loading

KristofferC commented Nov 20, 2020

mileslucas commented Nov 24, 2020 •

edited

Loading

mileslucas commented Nov 24, 2020 •

edited

Loading

KristofferC commented Nov 24, 2020 •

edited

Loading

giordano commented Nov 24, 2020

KristofferC commented Nov 24, 2020

Type piracy? #8

Type piracy? #8

Comments

giordano commented Nov 20, 2020

giordano commented Nov 20, 2020

mileslucas commented Nov 20, 2020 • edited Loading

giordano commented Nov 20, 2020

giordano commented Nov 20, 2020

KristofferC commented Nov 20, 2020 • edited Loading

mileslucas commented Nov 20, 2020 • edited Loading

KristofferC commented Nov 20, 2020

mileslucas commented Nov 24, 2020 • edited Loading

mileslucas commented Nov 24, 2020 • edited Loading

KristofferC commented Nov 24, 2020 • edited Loading

giordano commented Nov 24, 2020

KristofferC commented Nov 24, 2020

mileslucas commented Nov 20, 2020 •

edited

Loading

KristofferC commented Nov 20, 2020 •

edited

Loading

mileslucas commented Nov 20, 2020 •

edited

Loading

mileslucas commented Nov 24, 2020 •

edited

Loading

mileslucas commented Nov 24, 2020 •

edited

Loading

KristofferC commented Nov 24, 2020 •

edited

Loading