Below are the results of some writing test runs using the different Spark to CosmosDB connector methods.

Below are the Cosmos DB configurations used:

• Partitioned Collection: 250,000 RUs, 50 partitions
  • Collection 1: customerfactors_part 110GB with 19.80M documents
  • Collection 2: customerfactors 462.44GB with 64.00M documents

Below are the Apache Spark configurations used:

• HDI Cluster: Spark 2.1 HDI 3.5 multi-node Spark cluster, 2 masters, 2 zookeepers, driver memory 10g
  • Cluster 1: 4 workers (8 cores, 28GB memory), 32 cores, 128GB memory
    • Config 1.1: executor: 7 cores, 22GB memory
    • Config 1.2: executor: 2 cores, 6GB memory
  • Cluster 2: 9 workers (8 cores, 56GB memory), 72 cores, 504GB memory
    • Config 2.1: executor: 7 cores, 22GB memory
  • Cluster 3: 14 workers (4 cores, 28GB memory), 56 cores, 392GB memory
    • Config 3.1: executor: 3 cores, 22GB memory

Data migration: reading from a CosmosDB collection and writing directly to another CosmosDB collection

import com.microsoft.azure.cosmosdb.spark.schema._
import com.microsoft.azure.cosmosdb.spark._
import com.microsoft.azure.cosmosdb.spark.config.Config

val readConfig = Config(Map(
    "Endpoint" -> "",
    "Masterkey" -> "",
    "Database" -> "writingtests",
    "Collection" -> "customerfactors",
    "PageSize" -> "2000000",
    "preferredRegions" -> "North Europe;",
    "SamplingRatio" -> "1.0"))

val readDf = spark.sqlContext.read.cosmosDB(readConfig)

val writeConfig = Config(Map(
    "Endpoint" -> "",
    "Masterkey" -> "",
    "Database" -> "writingtests",
    "Collection" -> "customerfactors_shadow", 
    "WritingBatchSize" -> "500",
    "preferredRegions" -> "North Europe;",
    "SamplingRatio" -> "1.0"))

val writtenDf = coll.write.cosmosDB(writeConfig) 

Below are the results of writing performance on various setups.