[BUG] Make/Break Link Distributed Runtime

[nanwob]

hello,

i'm looking to create a fairly large scale network with CORE (800 nodes or so). Each node/container has 5 interfaces (i.e. eth0-eth4). Because these "links" are fairly stable, the plan is to use one of the "wired" links and then pre-calculate the physical connectivity topology and make/break physical links between all the nodes in real-time every few seconds.

If i want to update links and connectivity in real-time/run-time, is there a good example to follow on how to do this?

I found --
session.update_link(n1_id, n2_id, iface1_id, iface2_id, options)

Does this work across distributed machines?

[bharnden]

You could make/break links using session.update_link() by setting loss to 100, that would be one way. Otherwise you could physically remove and add links, but that would destroy and recreate interfaces.

If you are directly coding a session you could also just run commands to bring down targeted interfaces using node.cmd().

[nanwob]

Thanks! In thinking about this a bit more, i'm not sure if update_link() or node.cmd() (as i understand it) would work. Ideally, inside the container, i would like to have 5 fixed interfaces (eth0-eth4). However, these interfaces MAY be connected to 10-20 neighbors. I would like to choose which of the 10-20 neighbors (and their respective eth0-eth4) to connect to.

So i think I need something outside of the container to manage topology. I think this is analogous to the WLAN interface where everything comes out to a bridge and then eb_tables is used to managing who can actually talk to who. A few issues with WLAN is that 1) it connects to everyone "in range" so i cannot limit the degree of connectivity (i.e. connect to only these 5 nodes and ignore the rest) and 2) WLAN cannot go across distributed host machines. I could use EMANE, but EMANE has been known to be heavy-weight so i'm not sure how scalable it would be to 800 containers.

Given these constraints -- is there any way to achieve what we're looking for? Thanks!

[siliconja]

Hey Bow-Nan! I wonder if your use case would be best suited by EMANE's CommEffect model, which lends itself well to scripted events. I'm not sure how that would affect your scalability though, since you're talking about lots of nodes.

[bharnden]

WLAN has the potential to be controlled as you described by way of the gRPC API, alternatively you could refer to the server code to replicate this case. As long as the nodes do not move that will not trigger the nodes to connect/reconnect based on the WLAN range distance.

As far as the distributed part of the, I am sure there must be a way to bandage things to work across Hosts, I just haven't had a chance to dive into that.

Ebtables itself is also being deprecated and replaced by nftables in recent versions of distros, potentially when we need to switch we could take into account this case formally.

[nanwob]

Hi Jeff! Long time -- hope you are doing well! I forgot about Comm Effects! Thanks for the reminder -- yes that's exactly the interface i'm hoping to emulate. As you appropriately mentioned, the biggest thing is how it will scale. One question -- does CORE spawn a separate EMANE instance inside a container for each comm effects interface? Or is it a single EMANE instance that handles multiple interfaces? I'm guessing the former would be much more resource intensive.

@bharnden -- thanks for the tips and the reference to the code. Consider this a formal feature request to add distributed functionality to WLAN when you move away from EBTables. Thanks!!

[bharnden]

Sorry for the delay in responding. It is a single EMANE instance for all nems on a given node. We may look to break that out into a 1 to 1 in the future, but there will likely be a lot of loose ends that would need to be tied up to make that work.

Due to how EMANE works now, runtime add/deletion of links is not supported as would be desired, if the changes to break out EMANE into singular instances, that would make that possible.