IPv8 trackers often respond with an empty wan_introduction_address

[synctext]

No description provided.

[egbertbouman]

Currently, when we send an introduction request to an IPv8 tracker, the tracker will only respond with a non-empty wan_introduction_address for 30% of the time. The old Dispersy trackers, however, have this field populated 90+% of the time (see table below). This could slow down the bootstrapping process.
Of course, receiving an empty wan_introduction_address doesn't mean our entire request was in vain since the tracker will note our IP and will send it to other peers that are requesting peers for bootstrapping.

DiscoveryCommunity (7e313685c1912a141279f8248fc8db5899c5df5a)
100 introduction-requests, 10s apart

Tracker                 Responses    Non-empty introduction    Unique introductions
--------------------  -----------  ------------------------ -----------------------
130.161.119.206:6421          100                        93                      29
130.161.119.206:6422          100                        93                      31
131.180.27.155:6423           100                        90                      28
131.180.27.156:6424           100                        95                      33
131.180.27.161:6427           100                        29                      16
131.180.27.161:6521           100                        29                      16
131.180.27.161:6522           100                        30                      19
131.180.27.162:6523           100                        32                      19
131.180.27.162:6524           100                        31                      19
130.161.119.215:6525          100                        30                      19
130.161.119.215:6526          100                        30                      18
130.161.119.201:6527          100                        30                      18
130.161.119.201:6528          100                        30                      18

Another related issue is that when the IPv8 trackers do return a wan_introduction_address, it is often the same IP for all trackers. This likely means that IPv8 trackers have very few peers in their Network. I think that ideally, IPv8 trackers would keep a much larger cache of active peers.

[ichorid]

I've noticed this while testing Channels 3.0 prototype. Only one in three attempts to connect two peers in a custom community would work on average.

[qstokkink]

Precondition: All of these trackers are should be running the exact same script.

If the precondition holds, this is probably due to the flat 2 minute timeout in the tracker scripts.

We could increase the timeout if the trackers know only a small amount of peers. However, due to the hardware NAT puncturing timeout, the success rate of IntroductionRequests may then decrease as the firewall/router mappings close.

[egbertbouman]

@qstokkink Yes, this is why I didn't suggest just increasing the 120s removal time. We only want to gossip peers that the tracker still can connect to.

[qstokkink]

@egbertbouman if I read between the lines, it seems to me like concrete suggestion of this issue is then to add a keep-alive message to the bootstrap script (at least when the peer count is low). Is that correct?

[ichorid]

add a keep-alive message to the bootstrap script (at least when the peer count is low).

Sounds like a good idea. Seriously.

[egbertbouman]

@qstokkink That's correct. Although we also need to consider that peers may decide not to respond to an introduction request if the max_peers limit is reached. If that's the case, the tracker should also stop gossiping the peer even if it is still online. So, this keep-alive should probably also show if a peer is still looking for new peers (or we just don't respond to the keep-alive).

[qstokkink]

Sounds good. Thanks for clarifying 👍

[egbertbouman]

This previously mentioned keep-alive mechanism is added in Tribler/py-ipv8#954. Hopefully, this will fix the low number of introduction addresses that we get from trackers.

I also made a Jenkins job that gives some information about the wan_introduction_address/lan_introduction_address that we get from the live trackers. I plan on extending the graph with additional metrics in the future.

[egbertbouman]

I extended the Jenkins experiment a bit to show the number of reachable addresses. It turns out that a peer that does 100 bootstrap attempts at a >10s interval typically gets around 10 unique reachable peers from a tracker.

[synctext]

Could we please document the exact timeout and tracker behaviour in readthedoc?

[egbertbouman]

@synctext Sure. And I guess we should probably also discuss the basics of introduction-requests/responses and NAT puncturing. I don't think this documented either (or at least not that I can easily find).

[qstokkink]

If you want documentation on the original 2011 magic values (120 second timeout etc.) you can reference this paper:

Halkes G, Pouwelse J. UDP NAT and Firewall Puncturing in the Wild. InInternational Conference on Research in Networking 2011 May 9 (pp. 1-12). Springer, Berlin, Heidelberg.

[qstokkink]

This issue seems to have been fixed along the way by the following PR and resource:

• Updated tracker script py-ipv8#954
• https://py-ipv8.readthedocs.io/en/latest/reference/peer_discovery.html

@egbertbouman (since you're assigned) can we close this?

[egbertbouman]

Sorry for the late response, but yes, I think we can close it.