Flare is an experiment in using ChatGPT to rewrite an existing open source project in a different language. In this case Apache Spark, which is written largely in Scala, to Go.
Read the initial conversation for an idea of how this started.
A follow up conversation was started around implementing the Spark Connect interface in Go.
Using DeltaLake from Go is a driving goal for this package. Rivian wrote the Delta Connector we're using here specifically to share a common logstore with Spark. That feels like a hack. We're hoping to remove Spark from the equation altogether. If this feels too heavy we could consider others, like the one supported by treeverse