Preproc/aggregate

[atrostan]

No description provided.

[atrostan]

PR #14 can be ignored - this PR fixes some issues with jar building that is present in PR 14, once we merge into main, I'll delete PR 14.

• add 1d, 2d, hybrid partitioners; driver
• add partitioning, preprocessing scripts that execute Compressor and Partitioner JARs programmatically;
• add README explaining how to build Compressor and Partitioner JARs using assembly;
• use Scala Either to differentiate compression, partitioning of weighted vs. unweighted edge lists;
• add compression tests; Partitioning tests remain to be done;
• add test resources for CompressorTest;
• add spark SQL, dataframes;
• parse paths of worker/partitions nodes from yml;
• add symmetric rmat, weighted email graph;
• add types for rdd row for mains, mirrors;
• add aggregation driver that unions partitioned edgelists,
• assign mains, mirrors to partitions, and saves to partition paths;
• add rdd caching to compression script;
• revise assembly merge strategy for uber-jar assembly;
• remove hardcoded spark config;
• pass spark config in preprocess script;
• add tagged neighbours for mains, mirrors rdds;
• modify entity managers to initialize using partition specific mains, mirrors;
• modify VertexEntity neighbors to include edge weights;
• tag edges' src, dests with type:
  • main -> main, main -> mrr, mrr -> main, or mrr -> mrr;
• reformat, autoindent partition.util;
• shutdown actorsystems in cluster sharding app;

Apologies for large PR - there's lots to merge.