Skip to content

darkjh/go-mapreduce

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

11 Commits
 
 
 
 
 
 
 
 
 
 

Repository files navigation

go-mapreduce

Simple map-reduce implementation in Golang

Limits

Lots of limits for the moment:

  • single input file
  • processes should be on the same machine for a shared local FS
  • does not tolerate worker failure
  • a single job at a moment

Examples

Word count

  • Run a master go run wc.go master localhost:7777
  • Run a worker go run wc.go worker localhost:7777 localhost:7778
  • Submit the job go run wc.go submit examples/bible.txt localhost:7777

Alternatively the job can be run sequentially in a single process:

`go run wc.go sequential examples/bible.txt`

Result file is in the same directory as the input

`examples/mrtmp.bible.txt`

TODOs

  • Allow multiple jobs in parallel
  • Decouple map and reduce function definition from worker, they should be part of the job definition and get submitted to workers
  • Try out how to make use of mesos to run workers

About

Simple map-reduce implementation in Golang

Resources

Stars

Watchers

Forks

Releases

No releases published

Packages

No packages published

Languages