Skip to content

KeymanHuang/gearpump

 
 

Repository files navigation

GearPump Build Status codecov.io

download

GearPump is a lightweight real-time big data streaming engine. It is inspired by recent advances in the Akka framework and a desire to improve on existing streaming frameworks.

The name GearPump is a reference to the engineering term “gear pump,” which is a super simple pump that consists of only two gears, but is very powerful at streaming water.

We model streaming within the Akka actor hierarchy.

Per initial benchmarks we are able to process 11 million messages/second (100 bytes per message) with a 17ms latency on a 4-node cluster.

For steps to reproduce the performance test, please check Performance benchmark

Design Document

There is a 20 pages technical paper on typesafe blog, with technical highlights https://typesafe.com/blog/gearpump-real-time-streaming-engine-using-akka

Introduction and Motivation

Check Introduction

Getting Started

Check guide Get Started

How to Build

1). Clone the GearPump repository

  git clone https://github.com/intel-hadoop/gearpump.git
  cd gearpump

2). Build package

  ## Please use scala 2.11
  ## The target package path: target/gearpump-$VERSION.tar.gz
  sbt clean assembly packArchive ## Or use: sbt clean assembly pack-archive

After the build, there will be a package file gearpump-${version}.tar.gz generated under target/ folder.

NOTE: The build requires network connection. If you are behind an enterprise proxy, make sure you have set the proxy in your env before running the build commands. For windows:

Set HTTP_PROXY=https://host:port
set HTTPS_PROXT= https://host:port

For Linux:

export HTTP_PROXY=https://host:port
export HTTPS_PROXT= https://host:port

After the build, the package directory layout looks like this: Layout

Concepts

Check guide Concepts

How to write a GearPump Application

Check guide Streaming Application Developer Guide

How to manage the cluster

Check Admin Guide

Maven dependencies

Check Maven settings

Further information

Contributors (time order)

Contacts:

Please use the google user list if possible. For things that are not OK to be shared in maillist, please contact: [email protected] [email protected] [email protected]

License

Licensed under the Apache License, Version 2.0: https://www.apache.org/licenses/LICENSE-2.0

Acknowledgement

The netty transport code work is based on Apache Storm. Thanks Apache Storm contributors.

Thanks to Jetbrains for providing IntelliJ IDEA Free Open Source License.

About

Lightweight real-time big data streaming engine

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Scala 89.4%
  • JavaScript 5.9%
  • HTML 2.9%
  • Java 1.1%
  • CSS 0.7%