Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

attach finalizer in mmap to the correct object #54210

Merged
merged 3 commits into from
May 27, 2024
Merged

Conversation

KristofferC
Copy link
Sponsor Member

Fixes #54128

@KristofferC KristofferC added the backport 1.11 Change should be backported to release-1.11 label Apr 23, 2024
@KristofferC
Copy link
Sponsor Member Author

@vtjnash, this seems to make Mmap tests fail on Windows:

Mmap                                             (2) |         failed at 2024-04-23T11:08:39.378
Error During Test at C:\buildkite-agent\builds\win2k22-amdci6-5\julialang\julia-master\julia-39a8092b06\share\julia\test\testdefs.jl:24
  Got exception outside of a @test
  LoadError: SystemError: opening file "C:\\Users\\julia\\AppData\\Local\\Temp\\jl_MNqlAn4hyl": Invalid argument
  Stacktrace:
    [1] systemerror(p::String, errno::Int32; extrainfo::Nothing)
      @ Base .\error.jl:185
    [2] open(fname::String; lock::Bool, read::Nothing, write::Nothing, create::Nothing, truncate::Bool, append::Nothing)
      @ Base .\iostream.jl:298
    [3] open
      @ .\iostream.jl:277 [inlined]
    [4] open(fname::String, mode::String; lock::Bool)
      @ Base .\iostream.jl:361
    [5] open(fname::String, mode::String)
      @ Base .\iostream.jl:360
    [6] open(::Base.var"#470#471"{String, Tuple{}}, ::String, ::Vararg{String}; kwargs::@Kwargs{})
      @ Base .\io.jl:408
    [7] open
      @ .\io.jl:407 [inlined]
    [8] write(::String, ::String)
      @ Base .\io.jl:489
    [9] top-level scope
      @ C:\buildkite-agent\builds\win2k22-amdci6-5\julialang\julia-master\julia-39a8092b06\share\julia\stdlib\v1.12\Mmap\test\runtests.jl:106
   [10] include

@KristofferC KristofferC mentioned this pull request Apr 25, 2024
59 tasks
jmert added a commit to jmert/UnixMmap.jl that referenced this pull request Apr 25, 2024
In Julia 1.11+ with the new `Memory` object, the memory is not actually
owned by the constructed `Array` and therefore the finalizer may
trigger too early when the underlying memory is transfered from an
`Array` to a different kind of owner (such as `IOBuffer`).

Fix this by conditionally checking for the Julia 1.11+ behavior, and
instead setting the finalizer on the `Memory`, if necessary.

Duplicates the improvement made to the Mmap stdlib's implementation:
JuliaLang/julia#54210
jmert added a commit to jmert/UnixMmap.jl that referenced this pull request Apr 25, 2024
In Julia 1.11+ with the new `Memory` object, the memory is not actually
owned by the constructed `Array` and therefore the finalizer may
trigger too early when the underlying memory is transfered from an
`Array` to a different kind of owner (such as `IOBuffer`).

Fix this by conditionally checking for the Julia 1.11+ behavior, and
instead setting the finalizer on the `Memory`, if necessary.

Duplicates the improvement made to the Mmap stdlib's implementation:
JuliaLang/julia#54210
@KristofferC
Copy link
Sponsor Member Author

@vtjnash, do you have an idea about the Windows CI mmap failures here?

@vtjnash
Copy link
Sponsor Member

vtjnash commented May 7, 2024

No idea. Seems to imply it isn't getting finalized (or finalized soon enough for it to be happy) but I didn't look at what that test expects of it to see if it was reasonable looking

@nhz2
Copy link
Contributor

nhz2 commented May 21, 2024

I tested this locally and the Mmap tests pass if I move the GC.gc() on line 103 out of the if block to right before the write.

finalize(m); m=nothing; GC.gc()
end
write(file, "Hello World\n")

I also needed to move the GC.gc() on line 339 out of the do block to right before the rm.

finalize(A); A = nothing; GC.gc()
end
rm(file)

I have no idea why GC.gc() isn't calling all the finalizers when it's in an if block, but here is a simpler test I made for this:

if true
    foo = Ref(true)
    bar = Ref(foo)
    finalizer(x->x[][]=false, bar)
    bar=nothing
    GC.gc()
end
@info foo[]

This results in true, but moving the GC.gc() out of the if block:

if true
    foo = Ref(true)
    bar = Ref(foo)
    finalizer(x->x[][]=false, bar)
    bar=nothing
end
GC.gc()
@info foo[]

results in false

jmert added a commit to jmert/UnixMmap.jl that referenced this pull request May 21, 2024
In Julia 1.11+ with the new `Memory` object, the memory is not actually
owned by the constructed `Array` and therefore the finalizer may
trigger too early when the underlying memory is transfered from an
`Array` to a different kind of owner (such as `IOBuffer`).

Fix this by conditionally checking for the Julia 1.11+ behavior, and
instead setting the finalizer on the `Memory`, if necessary.

Duplicates the improvement made to the Mmap stdlib's implementation:
JuliaLang/julia#54210
@KristofferC
Copy link
Sponsor Member Author

Still seem to error on CI.

@nhz2
Copy link
Contributor

nhz2 commented May 22, 2024

Adding a GC.gc() on line 105 should fix it.

@KristofferC
Copy link
Sponsor Member Author

That seems to have done it. Thanks for the help @nhz2

@KristofferC KristofferC merged commit a1e1a3a into master May 27, 2024
7 checks passed
@KristofferC KristofferC deleted the kc/finalizer_mmap branch May 27, 2024 12:59
KristofferC added a commit that referenced this pull request May 27, 2024
KristofferC added a commit that referenced this pull request May 28, 2024
Backported PRs:
- [x] #53665 <!-- use afoldl instead of tail recursion for tuples -->
- [x] #53976 <!-- LinearAlgebra: LazyString in interpolated error
messages -->
- [x] #54005 <!-- make `view(::Memory, ::Colon)` produce a Vector -->
- [x] #54010 <!-- Overload `Base.literal_pow` for `AbstractQ` -->
- [x] #54069 <!-- Allow PrecompileTools to see MI's inferred by foreign
abstract interpreters -->
- [x] #53750 <!-- inference correctness: fields and globals can revert
to undef -->
- [x] #53984 <!-- Profile: fix heap snapshot is valid char check -->
- [x] #54102 <!-- Explicitly compute stride in unaliascopy for SubArray
-->
- [x] #54070 <!-- Fix integer overflow in `skip(s::IOBuffer,
typemax(Int64))` -->
- [x] #54013 <!-- Support case-changes to Annotated{String,Char}s -->
- [x] #53941 <!-- Fix writing of AnnotatedChars to AnnotatedIOBuffer -->
- [x] #54137 <!-- Fix typo in docs for `partialsortperm` -->
- [x] #54129 <!-- use correct size when creating output data from an
IOBuffer -->
- [x] #54153 <!-- Fixup IdSet docstring -->
- [x] #54143 <!-- Fix `make install` from tarballs -->
- [x] #54151 <!-- LinearAlgebra: Correct zero element in
`_generic_matvecmul!` for block adj/trans -->
- [x] #54213 <!-- Add `public` statement to `Base.GC` -->
- [x] #54222 <!-- Utilize correct tbaa when emitting stores of unions.
-->
- [x] #54233 <!-- set MAX_OS_WRITE on unix -->
- [x] #54255 <!-- fix `_checked_mul_dims` in the presence of 0s and
overflow. -->
- [x] #54259 <!-- Fix typo in `readuntil` -->
- [x] #54251 <!-- fix typo in gc_mark_memory8 when chunking a large
array -->
- [x] #54276 <!-- Fix solve for complex `Hermitian` with non-vanishing
imaginary part on diagonal -->
- [x] #54248 <!-- ensure package callbacks are invoked when no valid
precompile file exists for an "auto loaded" stdlib -->
- [x] #54308 <!-- Implement eval-able AnnotatedString 2-arg show -->
- [x] #54302 <!-- Specialised substring equality for annotated strs -->
- [x] #54243 <!-- prevent `package_callbacks` to run multiple time for a
single package -->
- [x] #54350 <!-- add a precompile signature to Artifacts code that is
used by JLLs -->
- [x] #54331 <!-- correctly track freed bytes in
jl_genericmemory_to_string -->
- [x] #53509 <!-- revert moving "creating packages" from Pkg.jl -->
- [x] #54335 <!-- When accessing the data pointer for an array, first
decay it to a Derived Pointer -->
- [x] #54239 <!-- Make sure `fieldcount` constant-folds for `Tuple{...}`
-->
- [x] #54288
- [x] #54067
- [x] #53715 <!-- Add read/write specialisation for IOContext{AnnIO} -->
- [x] #54289 <!-- Rework annotation ordering/optimisations -->
- [x] #53815 <!-- create phantom task for GC threads -->
- [x] #54130 <!-- inference: handle `LimitedAccuracy` in
`handle_global_assignment!` -->
- [x] #54428 <!-- Move ConsoleLogging.jl into Base -->
- [x] #54332 <!-- Revert "add unsetindex support to more copyto methods
(#51760)" -->
- [x] #53826 <!-- Make all command-line options documented in all
related files -->
- [x] #54465 <!-- typeintersect: conservative typevar subtitution during
`finish_unionall` -->
- [x] #54514 <!-- typeintersect: followup cleanup for the nothrow path
of type instantiation -->
- [x] #54499 <!-- make `@doc x` work without REPL loaded -->
- [x] #54210 <!-- attach finalizer in `mmap` to the correct object -->
- [x] #54359 <!-- Pkg REPL: cache `pkg_mode` lookup -->

Non-merged PRs with backport label:
- [ ] #54471 <!-- Actually setup jit targets when compiling
packageimages instead of targeting only one -->
- [ ] #54457 <!-- Make `String(::Memory)` copy -->
- [ ] #54323 <!-- inference: fix too conservative effects for recursive
cycles -->
- [ ] #54322 <!-- effects: add new `@consistent_overlay` macro -->
- [ ] #54191 <!-- make `AbstractPipe` public -->
- [ ] #53957 <!-- tweak how filtering is done for what packages should
be precompiled -->
- [ ] #53882 <!-- Warn about cycles in extension precompilation -->
- [ ] #53707 <!-- Make ScopedValue public -->
- [ ] #53452 <!-- RFC: allow Tuple{Union{}}, returning Union{} -->
- [ ] #53402 <!-- Add `jl_getaffinity` and `jl_setaffinity` -->
- [ ] #53286 <!-- Raise an error when using `include_dependency` with
non-existent file or directory -->
- [ ] #52694 <!-- Reinstate similar for AbstractQ for backward
compatibility -->
- [ ] #51479 <!-- prevent code loading from lookin in the versioned
environment when building Julia -->
@KristofferC KristofferC removed the backport 1.11 Change should be backported to release-1.11 label May 28, 2024
DilumAluthge pushed a commit that referenced this pull request Jun 3, 2024
lazarusA pushed a commit to lazarusA/julia that referenced this pull request Jul 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

segfault when reading from mmap derived from a file that get closed(?)
3 participants