norns-job 一个快速开发大数据任务工具

• 大数据开发项目经常面临重复性变量初始化，外部配置管理复杂
• 任务数量日益变多，相同类型任务自由组合为单个任务执行
• 任务依赖关系变化，修改配置文件即可
• ETL 快速开发
• 任务运行状态监控
• 公司应用数量日益变多，支持自动化上线
• 任务需要按不同应用不同日期执行
• 应用分优先级执行
• 重跑数据指定或者跳过具体任务

简单示例

// 默认开发
object HelloWorld {
   
  def main(args: Array[String]): Unit = {
    val sparkConf = new SparkConf()
      .setAppName("HelloWorld")
      .setMaster("local[1]")
    val spark = SparkSession.builder.config(sparkConf).getOrCreate
    spark.sql("select * from db")
    spark.stop()
  }

}

// 使用 norns-job 开发
class SingleSparkJobExample extends SparkJob {

  override def run(): Unit = {
    sql("select * from db")
  }

}

• 配置内容支持动态替换context#data

• QQ 群：348527861
• email：[email protected]

Copyright [2019] [NornsLab]

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

https://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.