This repository has been archived by the owner on Jun 17, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 57
This is a research prototype of the Dryad and DryadLINQ data-parallel processing frameworks running on Hadoop YARN.
License
MicrosoftResearch/Dryad
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Folders and files
Name | Name | Last commit message | Last commit date | |
---|---|---|---|---|
Repository files navigation
Dryad This is a research prototype of the Dryad and DryadLINQ data-parallel processing frameworks running on Hadoop YARN. Dryad utilizes cluster services provided as part of Hadoop YARN to reliably execute distributed computations on a cluster of computers. DryadLINQ provides the LINQ programming model for distributed data processing and leverages Dryad for reliable execution. Dryad and DryadLINQ on YARN are still under active development. If you just want to run a sample DryadLINQ program, the simplest way to get started is to follow the instructions at http:https://microsoftresearchsvc.github.io/Dryad/?topic=html/e992fd94-c956-481d-82e6-dbdf45daa722.htm Requirements Visual Studio 2012 An Azure HDInsight 3.0 cluster or a Windows YARN cluster composed of x64 machines Building Dryad 1) Clone the Dryad git repository. 2) Set the DRYAD_HOME environment variable to binary path (bin\Debug or bin\Release) under the directory Dryad was cloned to. 3) Use Visual Studio to open The Dryad solution file (Dryad.sln) located in the root of the repository and build the solution. The dependencies will be fetched from NuGet. Notes The HDFS implementation in Dryad currently only supports text files, although Azure Blobs can be used to store any serialized data type.
About
This is a research prototype of the Dryad and DryadLINQ data-parallel processing frameworks running on Hadoop YARN.