This repository contains some Spark- and Hadoop-based applications that can be run on MANTL as well as CDH platform.
Below is the full list of applications.
Application Name | Short Description | Can be run on MANTL? | Can be run on plain CDH? |
---|---|---|---|
TeraSort | Spark-based TeraSort benchmark | YES, MANTL-DEMO | YES, CDH-DEMO |
TestDFSIO | Spark-based TestDFSIO benchmark | YES, MANTL-DEMO | YES, CDH-DEMO |
Application Name | Short Description | Can be run on MANTL? | Can be run on plain CDH? |
---|---|---|---|
Spark-based DistCP | Application to copy data from HDFS to HDFS, from Swift to HDFS and vice versa | YES, MANTL-DEMO | YES, CDH-DEMO |
File Aggregator | Application to merge small files on HDFS | YES, MANTL-DEMO | YES, CDH-DEMO |
Cassandra-to-HDFS | Application to download Cassandra table on HDFS | YES, MANTL-DEMO | YES, CDH-DEMO |
Application Name | Short Description | Can be run on MANTL? | Can be run on plain CDH? |
---|---|---|---|
Kafka Producer | Reads directory and send messages to Kafka topic | YES, MANTL-DEMO | YES, CDH-DEMO |
Kafka-to-HDFS | Reads messages from Kafka topic and writes them to HDFS | YES, MANTL-DEMO | YES, CDH-DEMO |
Kafka-to-Cassandra | Reads messages from Kafka topic and writes them to Cassandra | YES | NO, requires Cassandra installation |
Kafka-to-ES | Reads messages from Kafka topic and writes them to Emastic Search | YES | NO, requires ES installation |
Kafka-to-Multiple | Reads messages from Kafka topic and writes them to HDFS, Cassandra and Elastic Search, runs a parallel thread that counts amount of saved records in each destination | YES, MANTL-DEMO | NO, requires Cassandra and ES installation |
Kafka Monitor | Reads messages from Kafka topic on Shipped Central cluster (containing information about CPU/RAM utilization of different hosts) and writes alerts to Elastic Search if any of these metrics are beyond given thresholds | YES, MANTL-DEMO | NO, requires ES installation and is specific to ShippedCentral |