pollfd fix, round 2 #12237

vtjnash · 2015-07-21T02:26:15Z

this is a rewrite of the poll_fd, poll_file, and watch_file APIs to more directly expose the libuv API and eliminate the callback legacy interface. since FDWatcher objects are now refcounted, this also solves #4401.

this replaces #7275

ping: @Keno

this eliminates the callback interface to the filesystem poll functions, to be consistent with the changes to the rest of base additionally, this exposes the full libuv API for watch_file and poll_file, rather than returning inconsistent return values from these two functions this fixes FDWatcher to ensure that it is unique for any given fd, (to avoid a libuv assertion). it must be manually closed before closing fd, or the applications behavior is undefined.

vtjnash · 2015-07-21T02:28:06Z

cross-ref: this will resolve some of #10293

vtjnash · 2015-07-21T02:29:29Z

this closes #7840 (when merged)

vtjnash · 2015-07-21T02:44:15Z

base/stream.jl

@@ -595,6 +597,7 @@ function _uv_hook_close(t::Timer)
 unpreserve_handle(t)
 disassociate_julia_struct(t)
 t.handle = C_NULL
+ notify(t.cond)


this fix is the cause of the warning in the repl.jl test: the Timer callback on line 100 of test/repl.jl is supposed to accept an argument t

JeffBezanson · 2015-07-21T03:16:33Z

Awesome, this is a relief!

I believe the only piece of #10293 left is SingleAsyncWork?

tkelman · 2015-07-21T07:41:07Z

base/libc.jl

+end
+
+@unix_only _get_osfhandle(fd::RawFD) = fd
+@windows_only _get_osfhandle(fd::RawFD) = WindowsRawSocket(ccall(:_get_osfhandle,Ptr{Void},(Cint,),fd.fd))


this'll need to be qualified with Libc. now anywhere it's used, like base/stream.jl and base/mmap.jl

tkelman · 2015-07-21T09:53:38Z

test/pollfd.jl

@@ -1,52 +1,90 @@
 # This file is a part of Julia. License is MIT: http:https://julialang.org/license

-@unix_only begin


why did you remove this? msvcrt has _pipe but it has extra arguments: https://msdn.microsoft.com/en-us/library/aa298531(v=vs.60).aspx

the eventual goal is to get this test running on windows too

not a bad goal, but will need to account for different spellings/api of posix functions - unless libuv has wrappers for all of these, which I think is one of the things we've been meaning to do upstream?

also works-around an issue where libuv timers could return too early on windows by several milliseconds this provides a general-purpose socketpair implementation on windows, for use in the tests and otherwise (although uv_pipe_link is recommended for any case where you don't explicitly need a WSA2 SOCKET)

…ows] Libc module (also use them to provide better errors in the tests) these belong in Libc since the are the windows counterparts to errno/strerror which are documented and exported in Libc

vtjnash · 2015-07-24T13:39:04Z

bump. this passes CI everywhere now...

JeffBezanson · 2015-07-24T20:36:38Z

doc/stdlib/io-network.rst

- Monitor a file for changes by polling every `interval_seconds` seconds for `seconds` seconds. A return value of true indicates
- the file changed, a return value of false indicates a timeout.
+ Monitor a file for changes by polling every ``interval_seconds`` seconds until a change occurs or ``timeout_s`` seconds have elapsed.
+ The `timeout_s` should be a long interval; the default is 5.007 seconds.


Should say interval instead of timeout_s.

ViralBShah · 2015-07-25T03:58:03Z

Marking this as 0.4 - assuming this is is done and long standing, and we should get it in.

vtjnash · 2015-07-27T15:40:20Z

bump. i'll merge this later today.

fixes poll_fd

amitmurthy · 2015-07-28T02:30:07Z

Awesome. 👍

tkelman · 2015-07-29T11:41:34Z

travis failure here https://gist.github.com/tkelman/96b21860b35454115373, are we back to timings that are too picky for busy low-memory VM's?

amitmurthy · 2015-07-29T14:15:45Z

OTOH, in this case it does seem like it caught an actual bug. The poll call should not have returned before intvl seconds.

tkelman · 2015-07-31T03:52:47Z

Another failure: https://travis-ci.org/JuliaLang/julia/jobs/73427729

exception on 5: ERROR: LoadError: test failed: 0.1 <= 0.099903232
 in expression: intvl <= t_elapsed
 in yieldto at ./task.jl:24
 in wait at ./task.jl:313
 in wait at ./task.jl:228
 in wait at ./task.jl:61
 in sync_end at ./task.jl:340
 in anonymous at no file:348
 in include_string at loading.jl:158
 in include_from_node1 at ./loading.jl:200
 in runtests at /tmp/julia/share/julia/test/testdefs.jl:173
 in anonymous at multi.jl:909
 in run_work_thunk at multi.jl:648
 in anonymous at task.jl:909
while loading /tmp/julia/share/julia/test/pollfd.jl, in expression starting on line 65
    From worker 3:       * bitarray              in  90.72 seconds
    From worker 4:       * mpfr                  in   4.91 seconds
    From worker 2:       * subarray              in 353.43 seconds
ERROR: LoadError: LoadError: test failed: 0.1 <= 0.099903232
 in expression: intvl <= t_elapsed
 in anonymous at task.jl:1442
while loading /tmp/julia/share/julia/test/pollfd.jl, in expression starting on line 65
while loading /tmp/julia/share/julia/test/runtests.jl, in expression starting on line 5
    From worker 5:       * pollfd

tkelman · 2015-08-11T19:03:36Z

another pollfd failure on travis https://s3.amazonaws.com/archive.travis-ci.org/jobs/75113270/log.txt

and https://s3.amazonaws.com/archive.travis-ci.org/jobs/75295368/log.txt

timholy · 2015-08-12T13:58:33Z

Another pollfd failure on AppVeyor:

in #12560.

vtjnash · 2015-08-12T18:50:29Z

i'm not really sure what that test is doing in the pollfd.jl file, since it is testing the functionality of one-shot Timers, not any of this code

@JeffBezanson @Keno is 0.2 sec sometimes not enough for a 0.01 sec timer to trigger? (added in 7f4c2de)

tkelman · 2015-08-13T19:15:13Z

same Any[] == [1] pollfd failure on a buildbot http:https://buildbot.e.ip.saba.us:8010/builders/build_ubuntu14.04-x86/builds/2163/steps/shell_2/logs/stdio

vtjnash · 2015-08-19T20:37:59Z

latest travis failure on #12463 (https://travis-ci.org/JuliaLang/julia/builds/76174785) was also due to Jeff's new test for Timers there

amitmurthy and others added 3 commits July 20, 2015 14:42

revised poll_fd tests

1f27d75

standardized event property testing

4555d49

vtjnash added the domain:io Involving the I/O subsystem: libuv, read, write, etc. label Jul 21, 2015

vtjnash reviewed Jul 21, 2015
View reviewed changes

tkelman reviewed Jul 21, 2015
View reviewed changes

Qualify Libc._getosfhandle

9b789b8

tkelman reviewed Jul 21, 2015
View reviewed changes

vtjnash force-pushed the jn/pollfd_fix2 branch from 031feba to 639f7e8 Compare July 22, 2015 07:40

vtjnash added 2 commits July 22, 2015 13:29

update pollfd docs based on reviewer comments

84e969a

vtjnash force-pushed the jn/pollfd_fix2 branch from 639f7e8 to 84e969a Compare July 22, 2015 17:56

provide GetLastError/FormatMessage as official functions of the [Wind…

64c184b

…ows] Libc module (also use them to provide better errors in the tests) these belong in Libc since the are the windows counterparts to errno/strerror which are documented and exported in Libc

JeffBezanson reviewed Jul 24, 2015
View reviewed changes

ViralBShah added this to the 0.4.0 milestone Jul 25, 2015

fix poll_file variable name in doc

99b56cd

vtjnash added a commit that referenced this pull request Jul 27, 2015

Merge pull request #12237 from JuliaLang/jn/pollfd_fix2

39bd1fc

fixes poll_fd

vtjnash merged commit 39bd1fc into master Jul 27, 2015

vtjnash deleted the jn/pollfd_fix2 branch July 27, 2015 16:10

tkelman mentioned this pull request Jul 28, 2015

Assertion failure in libuv #7840

Closed

tkelman mentioned this pull request Aug 27, 2015

pollfd test fails for small intervals #12824

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pollfd fix, round 2 #12237

pollfd fix, round 2 #12237

vtjnash commented Jul 21, 2015

vtjnash commented Jul 21, 2015

vtjnash commented Jul 21, 2015

vtjnash Jul 21, 2015

JeffBezanson commented Jul 21, 2015

tkelman Jul 21, 2015

tkelman Jul 21, 2015

vtjnash Jul 21, 2015

tkelman Jul 21, 2015

vtjnash commented Jul 24, 2015

JeffBezanson Jul 24, 2015

ViralBShah commented Jul 25, 2015

vtjnash commented Jul 27, 2015

amitmurthy commented Jul 28, 2015

tkelman commented Jul 29, 2015

amitmurthy commented Jul 29, 2015

tkelman commented Jul 31, 2015

tkelman commented Aug 11, 2015

timholy commented Aug 12, 2015

vtjnash commented Aug 12, 2015

tkelman commented Aug 13, 2015

vtjnash commented Aug 19, 2015

		@@ -1,52 +1,90 @@
		# This file is a part of Julia. License is MIT: http:https://julialang.org/license

		@unix_only begin

pollfd fix, round 2 #12237

pollfd fix, round 2 #12237

Conversation

vtjnash commented Jul 21, 2015

vtjnash commented Jul 21, 2015

vtjnash commented Jul 21, 2015

vtjnash Jul 21, 2015

Choose a reason for hiding this comment

JeffBezanson commented Jul 21, 2015

tkelman Jul 21, 2015

Choose a reason for hiding this comment

tkelman Jul 21, 2015

Choose a reason for hiding this comment

vtjnash Jul 21, 2015

Choose a reason for hiding this comment

tkelman Jul 21, 2015

Choose a reason for hiding this comment

vtjnash commented Jul 24, 2015

JeffBezanson Jul 24, 2015

Choose a reason for hiding this comment

ViralBShah commented Jul 25, 2015

vtjnash commented Jul 27, 2015

amitmurthy commented Jul 28, 2015

tkelman commented Jul 29, 2015

amitmurthy commented Jul 29, 2015

tkelman commented Jul 31, 2015

tkelman commented Aug 11, 2015

timholy commented Aug 12, 2015

vtjnash commented Aug 12, 2015

tkelman commented Aug 13, 2015

vtjnash commented Aug 19, 2015