nsqd: add topology region/zone aware message consumption

[jehiah]

This updates nsqd to have a --topology-region and --topology-zone config argument and prefer sending messages to same-zone and same-region consumers as proposed in #1300

[jehiah]

I think this is ready for a first round of review @mreiferson @ploxiln (along w/ it's pair nsqio/go-nsq#312

Some items to decide on -

• Should -mem-queue-size=0 disable zone local and region local consumption? (i.e. should it continue to effectively write to disk first?)
• Is there an approach to changing the consumption of the disk read chan so it prefers zoneLocal and regionLocal consumers? (this would probably mean a goroutine that consumes it and does put() instead of messagePump consuming the disk backend directly. If there is an approach is it important to do as part of this?

Once this is squared away and we are happy with it i'll follow up with documentation PR's and exposing region/zone in nsqadmin.

[ploxiln]

I had some ideas, for what I think are minor simplifications that leave the overall design the same. Overall looks good to me.

[ploxiln]

should probably mention something about "for preferring closer consumer"

[zoemccormick]

@jehiah @ploxiln @mreiferson This PR is now officially ready for review - in conjunction with nsqio/go-nsq#312 and nsqio/nsqio.github.io#89.

We have written up an experience report based on our observations running these changes for 2 months in our production environment - it can be found here. It also lays out the changes to nsqadmin.

Please let me know if you have any other questions! Thanks in advance!

[mreiferson]

@jehiah hello! what's the plan here for actually landing this? Are y'all waiting on me to review everything 😏?

[jehiah]

@mreiferson sorry for the radio silence on this - I think we are ready to give more attention to this in Q4 so if you have any comments on this PR (or the related go-nsq changes) in the next few weeks please share them otherwise the topology changes are ready to land and we'll do that next month.

I think the general plan is to still land this behind the feature flag and get them in a 1.4 release and consider removing the FF in 1.5. We have been happy with this running at Bitly for a while now, but i'm keeping an eye on kubernetes/enhancements#4747 as it will make adopting in a K8s environment easier.