Implement UDP

[Keno]

@StefanKarpinski I figured it'd be faster if I did it myself. This needs some more tests and documentation (if you wouldn't mind taking care of that), but other than that this is good to go.

[vtjnash]

This looks strange. UDP is an acronym so shouldn't it be all caps UDPSocket. (also, why is it TcpServer, not TCPServer, etc?)

[johnmyleswhite]

+1 for more consistent use of caps

[Keno]

Not sure, I did it for consistency with TCP, I'd vote for leaving this as is for this PR and then if we decide to rename, we can do that in one commit for both.

[staticfloat]

+1 for TCPxxx and UDPxxx, although that will be a breaking change.

[vtjnash]

I think you can use int(nread) here and avoid the extra copy

[Keno]

Yes, I suppose that's a good idea.

[G3UKB]

As I started this I guess I should give some input. I'm no expert, just a user, but having used this protocol for the same application in a few different languages (last one was go) here is what I would need. I've added the relevant code in go syntax.

1. Listen on all adapters.
   udpAddr, err := net.ResolveUDPAddr("up4", ":0")
   conn, err = net.ListenUDP("udp", udpAddr)
2. Set buffers. This is essential to stop overflow (real-time data) and be able to buffer a number of messages. Messages are 1032 bytes and I want to block on writing.
   conn.SetReadBuffer(103200)
   conn.SetWriteBuffer(1032)
3. I then need to send a broadcast message.
   dst, err := net.ResolveUDPAddr("udp", "255.255.255.255:1024")
   conn.WriteTo(msg, dst)
4. Finally listen for broadcast responses with a timeout.
   set_timeout(conn, 5e9)
   n, peer, err := conn.ReadFrom(resp)
5. I'm then into straight forward data exchanges reading, processing and outputting 1032 byte messages.

I would just add that I believe its normal to do ones own management so I would not for example expect an error to be thrown if a packet is dropped nor get partial packets. I have sequence numbers in all packets so I detect dropped and out of sequence packets. I don't know if this is standard but it seems logical to me on an unreliable connection.

--bob

[StefanKarpinski]

That is informative. I would still like to try raising an error by default and seeing how that goes. I suspect this doesn't happen much in practice.

[Keno]

Alright, let's leave it like this then.

[bass3m]

Any updates on when this will get merged ? For what it's worth, i've been playing around with @loladiro's patch and it works great.

[Keno]

As soon as somebody writes documentation and a few more tests. I'll get to it eventually, but am quite busy at the moment. Feel free to tackle that though.

[bass3m]

@loladiro sounds good, i'll see what i can do for some tests and documentation.

[StefanKarpinski]

Shall we merge it and open an issue for documentation and testing? Otherwise someone would have to make a pull request against your pull request...

[Keno]

Shall we merge it and open an issue for documentation and testing?

Why do I have a feeling that questionmark was rhetorical? ;)

[StefanKarpinski]

Yeah, I realized right after posting that I should just do it instead of suggesting it and waiting for someone else to do it.

[Keno]

That's how things get done.

[vtjnash]

I would do that with #3796, but somehow I feel that's how things get un-done. Unless someone else was to merge it, like Stefan did here...

[StefanKarpinski]

That's a little different – #3796 might break everything. This is quite safe as long as you don't use the new functionality and there's at least one report that the new functionality works nicely.

[StefanKarpinski]

But I am somewhat inclined at this point to just merge it and see what happens.

[bass3m]

Thanks for merging it, this will make things easier.

[danluu]

Looks like I'm late to the party here, but I wanted to echo @Bob-C's comment above. I'm not a networking expert, but I've spent some time with UDP and implementing protocols on top of UDP, and I find it pretty surprising that the UDP API throws an exception on a partial packet instead of just dropping it.

Unless you have what is, by non-HPC standards, a fairly exotic setup, packets are expected to get dropped all the time, and throwing an exception on some drops doesn't seem particularly helpful. And if if you have a lossless fabric with a setup that isn't expected to drop anything, it would be pretty unorthodox to use libuv or something that wraps libuv.