support multiple topologies

[amitmurthy]

edit : current iteration : #11665 (comment)

Currently all workers are connected to all other workers. However, a large number of use cases require connections between the master and workers only, and the we can optimize the total number of TCP connections for such cases, especially with hundreds of workers.

This PR provides a means of supporting multiple connection topologies in the future. Currently ALL_TO_ALL and MASTER_SLAVE are supported.

Default is MASTER_SLAVE, which is a BREAKING change for programs that expect all to all connections.

With all_to_all, @time addprocs(8) takes around 1.7 seconds.
With master_slave, @time addprocs(8) takes around 0.8 seconds.

• program arg -I or --interconnect specifies the interconnect type to be setup by addprocs
• valid options are all_to_all, master_slave and custom. Default is master_slave
• custom indicates that the cluster manager is responsible for the connection topology and it will provide the list of workers that a worker must connect with. Currently unsupported.
• the master continues to be connected to all workers.
• We cannot have a mix of workers with some connected via all_to_all and the rest in master_slave mode.

While currently we cannot send requests between nodes that are not connected to each other, we can build on this to implement routing mechanisms that will efficiently transport messages via intermediate, mutually connected nodes. Especially required for broadcast type of requests.

edit : standardized on the term "topology" in place of "interconnect".

[StefanKarpinski]

There seem to be two different terms in play here: "interconnect" and "topology". I find "topology" to be much clearer and more obvious – can we just use that one consistently throughout? In either case, I feel like we should use just one of these two terms consistently.

[amitmurthy]

My rationale for choosing the term "interconnect" over "topology" is that I tend to associate plain "topology" with the physical topology of the cluster. While "interconnect" here is the logical topology of the interconnections.

For example, all workers, each with the same physical network characteristics may opt for a master_slave only interconnect model since a) the algorithm does not require any worker-worker communication, b) startup time is faster and c) reduces unnecessary tasks and socket in the workers.

But this may just be me. If the meaning of "topology" is widely used to describe both physical and logical descriptions of interconnections, I can change over to the same.

[andreasnoack]

Is it worth/necessary exporting three new names to have this new option? They are only used in set_interconnect, rught? So maybe just have set_interconnect(s::Symbol) with :master_slave, :all_to_all and :custom and then have the enums unexported. I'm also wondering if it is enough to call the function interconnect (or topology) and avoid the underscore.

[amitmurthy]

Sort of defeats the purpose of enums, no? We could still unexport it and require usage to be Multiprocessing.IC_MASTER_SLAVE or Parallel.IC_MASTER_SLAVE once #11638 is merged.

interconnect is fine.

[andreasnoack]

I haven't used enums in Julia yet so I haven't thought much about what is the right way to use them. It just seems a bit much to export new names when they are only used in his function and at the same time Multiprocessing.IC_MASTER_SLAVE or Parallel.IC_MASTER_SLAVE is a bit long.

[StefanKarpinski]

I really think that the term "topology" is less awkward than "interconnect" here.

[ViralBShah]

IMO the term topology is usually used in the logical sense. Eg. in MPI:
https://computing.llnl.gov/tutorials/mpi/#Virtual_Topologies

Whereas interconnect is used for physical network layer - like Infiniband interconnect. Eg:
https://en.wikipedia.org/wiki/InfiniBand

[StefanKarpinski]

That seems to qualify as a vote to call this "topology", no?

[ViralBShah]

Yes - my vote is to use topology as the term for this work. Also, I think this is really cool - which got trumped in the discussion on terminology!

[StefanKarpinski]

Indeed – this is great to have – and absolutely crucial for high performance distributed computing.

[JeffBezanson]

I totally agree with Viral on terminology, and that this is awesome :)

[tkelman]

+1 for topology, and for #11665 (comment)

[alanedelman]

Just my own two cents .... nothing in computer science is ever defined of course

Interconnect ... for me is often about the wiring and perhaps the networking
e.g. ethernet with infiniband but not so much the graph theory

topology ..... is what one might call the graph theory
e.g. hypercube, ring, master-slave, star, random, a big mess

topology could be instantiated in wires, in communications software or in algorithms
when it's not physical wires it may be called virtual topology

Being a math geek, i like the word "topology" too

[amitmurthy]

OK. "topology" it is.

[amitmurthy]

This PR has the first steps towards a complete topology support:

• a new keyword arg to addprocs, topology that specifies if we want all_to_all, master_slave only or a custom defined topology. all_to_all is the default.
• Custom topologies are specified via the ClusterManager interface.
• The changes to existing all_to_all connection setup is minimal.
• The feature has been marked as experimental. Currently messages between unconnected workers results in an error. This will change in the future when we route via intermediate nodes. Much work in auto-detecting actual physical network topologies and setting up connections accordingly needs to be done.

The arguments for merging this now instead of waiting for 0.5 are

• master_slave only connections for programs not requiring all_to_all connections will
  • speed up addprocs
  • optimize system resources w.r.t number of open file fds and TCP connections
• The changes are minimal and does not break any existing behavior
• feedback on this feature during stable 0.4 phase will be useful for a complete implementation.

[amitmurthy]

I'll rebase and merge this in 2-3 days if there are no issues.

[amitmurthy]

Merging this once CI is green.

[glesica]

Is there a reason this feature is undocumented in 0.4.2? It is quite helpful!

[amitmurthy]

The documentation is sparse, but it is there - http:https://docs.julialang.org/en/release-0.4/manual/parallel-computing/#specifying-network-topology-experimental

[glesica]

Ahh, I was looking at the REPL docs for addprocs. I didn't see it listed.

[amitmurthy]

Yes, we should incorporate it in the stdlib docs too.

[Ismael-VC]

Also in NEWS.md would be nice, this is great!