Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature] Include the Apache DolphinScheduler on Amazon EKS solution into unified repository maintenance #621

Open
SEZ9 opened this issue Aug 27, 2024 · 0 comments
Labels
enhancement New feature or request

Comments

@SEZ9
Copy link

SEZ9 commented Aug 27, 2024

Community Note

  • Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request
  • Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request
  • If you are interested in working on this issue or have submitted a pull request, please leave a comment

What is the outcome that you are trying to reach?

  1. Apache DolphinScheduler is an Apache top-level project. It's a distributed and extensible open-source workflow orchestration platform with powerful visual DAG interfaces, offering convenience in visual organization. It has a broad global AWS customer base.
  2. A more efficient architecture and performance. It natively supports the visual invocation of various AWS components, including data services like Redshift, Athena, Glue, DMS, and EMR.
  3. By integrating with AWS EKS, Aurora database, and others, it provides a more reliable and scalable architecture. Of course, many customers are already using Apache DolphinScheduler on AWS EKS.

Describe the solution you would like

A Terraform script has been created by AWS SA to deploy Apache DolphinScheduler to EKS with a single click. It supports EFS persistence, remote Task log writing to S3, and Aurora Serverless database for metadata storage .

Describe alternatives you have considered

Currently, Airflow does not support visual task orchestration, and its integration with AWS native services is not as convenient as Apache DolphinScheduler. Therefore, integrating Apache DolphinScheduler is highly necessary.

Additional context

In the past, we have had close collaboration with the Apache DolphinScheduler open-source community, publishing multiple joint blogs, including technical best practices and customer case studies.

  1. How to Integrate Apache DolphinScheduler with AWS EMR&Redshift
  2. Using Apache DolphinScheduler for Amazon EMR Job Scheduling
  3. Using Amazon EMR Serverless, Athena, and Dolphin Scheduler to build an on-cloud and off-cloud data synchronization solution
  4. Deploy the serverless Apache DolphinScheduler task scheduling system on AWS
@askulkarni2 askulkarni2 added the enhancement New feature or request label Aug 27, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

2 participants