ashao: HW9 commit

[ashao]

No description provided.

[jakevdp]

This is an interesting approach – using a DirectView and scattering periods to the available clusters. All the other solutions I've seen have used a LoadBalancedView and pushed each individual problem to a single core. It's not obvious to me which one would be faster... but I like this solution!

[ashao]

Hi Jake,
I thought about that too and a priori I think that the LoadBalancedView way
is probably faster because there's less communication overhead (though
presumably that's negligible on a shared-memory machine). One question that
I haven't quite been able to figure out via Google is whether there's a
threaded version of Numpy/Scipy or if using this builtin iPython way of
parallelizing routines is the best way to perform operations on large
amounts of data?

Thanks!
Andrew

On Mon, Dec 8, 2014 at 9:07 AM, Jake Vanderplas [email protected]
wrote:

Merged #159 #159.

—
Reply to this email directly or view it on GitHub
#159 (comment).

[jakevdp]

Hi – there's not any general threaded version of numpy, so with IPython you're mostly stuck with doing embarrassingly parallel approaches that don't require data sharing between processes. One way you can get around this is to use memory-mapped arrays. There's a bit of info on this in Olivier Grisel's parallel machine learning tutorial: https://github.com/ogrisel/parallel_ml_tutorial