reworked PartionedSerializer (now DestinationSerializer)

[amitmurthy]

• Rename PartionedSerializer to DestinationSerializer
• Make the destination aware serializer more generic
• Fix bug w.r.t. pids having the correct shape. We should ensure that the old PartionedSerializer implementation of DArrays does not make it to METADATA for Julia 0.4.

[amitmurthy]

@andreasnoack , the interface now is to provide a closure of the form pididx -> data_to_be_serialized_to_pid where pididx is a linear index into the pids supplied as part of the DestinationSerializer constructor. The closure can capture all information required to return the subset of data to be serialized.

Feedback?

[andreasnoack]

Trying to wrap my head around this. Would it be possible to hide this functionality behind a serialize method for SubArray?

[amitmurthy]

Serializing a view already serializes only the subarray part - I just tried it with an IOBuffer and a serialize/deserialize on a view. The issue is how to identify which subarray to serialize depending on the destination worker.

[andreasnoack]

I getting closer but I'm still not there. If we compare a DestinationSerializer based solution to the version of scale! below, would the DestinationSerializer be able to only serialize each chunk of x once?

function Base.LinAlg.scale!(x::AbstractVector, A::DMatrix)
    asyncmap(procs(A)) do p
        remotecall_fetch(p) do
            scale!(Array(view(x, localindexes(A)[1]...), localpart(A))
            nothing
        end
    end
    A
end

[amitmurthy]

Something like this?

function Base.LinAlg.scale!(x::AbstractVector, A::DMatrix)
    ds = DestinationSerializer(procs(A)) do pididx
        x[A.indexes[pididx][1]]
    end

    asyncmap(procs(A)) do p
        remotecall_fetch(p) do
            scale!(localpart(ds), localpart(A))
            nothing
        end
    end
    A
end

If A is distributed over its first dimension, then only the parts of x relevant to process p would be serialized to p. The DestinationSerializer is not about serializing x only once - it is about serializing only that part of x as required on worker p.

I have also renamed verify_and_get to localpart as it is in fact only that part of the object as required on p.

[amitmurthy]

A.indexes is the array of all indices on all workers. It has the same size as A.pids, i.e., procs(A)

[vchuravy]

@amitmurthy Can you rebase onto master so that we don't get test failures from the nightly?

[amitmurthy]

Done.

[amitmurthy]

OSX error appears unrelated.