make nprocs report only fully connected workers

[tanmaykm]

This changes nprocs (and therefore nworkers) to report only workers in W_CONNECTED state.

This is to avoid failures in certain conditions with @everywhere or other similar methods that broadcast
messages to all workers.

To simulate:

• introduce an artificial delay at

  julia/base/distributed/cluster.jl

  Line 443 in 2c4f6d7

  (r_s, w_s) = connect(manager, w.id, wconfig)

  with sleep(10)
• start a master with:

using ClusterManagers

ElasticManager(;addr=IPv4("0.0.0.0"), port=9009, cookie="cookie", topology=:master_slave)

while nworkers() < 4
    sleep(1)
end

@everywhere println(myid())

• start 4 workers with:

using ClusterManagers
ClusterManagers.elastic_worker("cookie", "127.0.0.1", 9009; stdout_to_master=false)

Without this change, this will often result in:

ERROR: LoadError: peer 3 is not connected to 1. Topology : master_slave
check_worker_state(::Base.Distributed.Worker) at ./distributed/cluster.jl:115
send_msg_(::Base.Distributed.Worker, ::Base.Distributed.MsgHeader, ::Base.Distributed.CallMsg{:call_fetch}, ::Bool) at ./distributed/messages.jl:180
remotecall_fetch(::Function, ::Base.Distributed.Worker, ::Expr, ::Vararg{Expr,N} where N) at ./distributed/remotecall.jl:346
remotecall_fetch(::Function, ::Int64, ::Expr, ::Vararg{Expr,N} where N) at ./distributed/remotecall.jl:367
(::##1#3)() at ./distributed/macros.jl:84

[tkelman]

does anything under the TEST_FULL flag mock the conditions where this would differ, for testing?

[amitmurthy]

does anything under the TEST_FULL flag mock the conditions where this would differ, for testing?

No.

[amitmurthy]

@tanmaykm - The change is fine. Could you also fix procs() and workers() which suffer from the same issue in this PR?

[tanmaykm]

Yes, will do that in a bit.