RFC: ByteOrder types for io

[simonbyrne]

This adds ByteOrder types and read/write methods to make it easier to deal with different byte orderings.

The key advantage is arrays: previously I found myself using code such as:

x = read(f,Float64,100)
map!(ntoh,x)

Now it is possible to use

x = read(f,Float64,NetworkByteOrder,100)

It still needs some tests, documentation and exports, but in the meantime I would appreciate any feedback on the interface.

[ViralBShah]

Cc: @loladiro

[Keno]

There needs to be a generic fallback that defaults to HostByteOrder

[Keno]

while you're at it, this is much more nicely done using reinterpret instead of box/unbox.

[diegozea]

I don't know... reinterpret for single values make a lot more of operations.

[Keno]

that's not true.

[Keno]

julia> function bar(x)
       Intrinsics.box(Float64,Intrinsics.unbox(Int64,x))
       end
# methods for generic function bar
bar(x) at none:2

julia> disassemble(bar,(Int64,))

define double @julia_bar621(i64) {
top:
  %1 = bitcast i64 %0 to double, !dbg !5031
  ret double %1, !dbg !5031
}

julia> function foo(x)
       reinterpret(Float64,x)
       end
# methods for generic function foo
foo() at none:2
foo(x) at none:2

julia> disassemble(foo,(Int64,))

define double @julia_foo600(i64) {
top:
  %1 = bitcast i64 %0 to double, !dbg !4869
  ret double %1, !dbg !4869
}

[diegozea]

Good :) I belief that reinterpretmakes conversion from value to array and get a value again by definition on Array. Based on function declaration at array.jl
Maybe it's something more to add into the style notes of the docs ;)

[Keno]

There is a pure-bits read/write method for arrays which will need to be adjusted.

[simonbyrne]

I've added BitArray and Complex methods (and added the reinterpret as requested).

As for the fallback: there's one for Number (which currently are the only methods affected): I tried to create something more general but I kept getting ambiguous dispatch warnings. The code is a bit messy at the moment: i.e. for arrays I have to define methods with and without the ByteOrder argument. If anyone has any suggestions on how to clean it up, I would be grateful.

[simonbyrne]

One other point: should HostByteOrder and ReverseByteOrder be the concrete types, and NetworkByteOrder and LittleByteOrder be aliases, or vice versa? The current choice was completely arbitrary, so if there is any reason to prefer one to the other, it can be easily changed.

[timholy]

This looks great. I'd say if you can resolve the merge conflict and perhaps write a few more tests (e.g., floating-point scalars, etc.), then we should merge this.

[pao]

I suppose if this is merged StrPack should switch to this type. (Previously, it had been changed from a type to symbols due to the language zeitgeist...is the pendulum swinging back?)

Of course would also take advantage of the updated method signatures.

[simonbyrne]

Okay, I've rebased to master, and added some more tests.

I'm still not sure about the interface

• I would have liked to use keyword arguments, but this would require everyone else to define their read/write methods with them, otherwise the Array methods would break (including cases such as Uint8 where they're not needed). Any ideas?
• The only other alternative I could think of was to define them as parametric types, i.e. write(io,NetworkByteOrder(4.0)), read(io,NetworkByteOrder{Float64}), but again this didn't seem to work for arrays.

I haven't really kept up with the current trends re: tags. The only other place in base where they seem to be widely used is sort.jl, but they do it slightly differently by exporting const instances. Would that be a better approach?

[timholy]

I think for low-level IO keyword args are not the right way to go. This leverages dispatch.

I think it's plenty good, and the issue about const instances can be addressed later if someone thinks it's important. Enough bikeshedding.

[StefanKarpinski]

I guess I'm a little late here, but I object to this interface. Primarily my objection is that byte order should be property of streams, not data, and certainly shouldn't normally be something that is specified on individual reads and writes. I'm also not super thrilled about the API of passing these network byte order objects as straggling arguments.

[simonbyrne]

Well, what about if we use a type wrapper around IO types? Something like:

type NetworkByteOrder{T<:IO}
  io::T
end
type LittleByteOrder{T<:IO}
  io::T
end

read(NetworkByteOrder(io),Float64)

[StefanKarpinski]

That seems much better to me. I think @JeffBezanson has some opinions here as well.