CPU OutOfMemory: could not split inputs and retry

[SidWeng]

The exception is thrown from executor, but the executor memory should be enough (50gb). Besides, I use top command to monitor the memory usage for all nodes, the usage is never higher than 50%. How do I prevent this OOM error or any way or tool can help me find the root cause ?

07:15:20.922 ERROR SupervisorStrategy - Job aborted due to stage failure: Task 32 in stage 5.0 failed 4 times, most recent failure: Lost task 32.3 in stage 5.0 (TID 277) (10.0.0.14 executor 0): com.nvidia.spark.rapids.jni.CpuSplitAndRetryOOM: CPU OutOfMemory: could not split inputs and retry
	at com.nvidia.spark.rapids.RmmRapidsRetryIterator$AutoCloseableAttemptSpliterator.split(RmmRapidsRetryIterator.scala:460)
	at com.nvidia.spark.rapids.RmmRapidsRetryIterator$RmmRapidsRetryIterator.next(RmmRapidsRetryIterator.scala:588)
	at com.nvidia.spark.rapids.RmmRapidsRetryIterator$RmmRapidsRetryAutoCloseableIterator.next(RmmRapidsRetryIterator.scala:517)
	at com.nvidia.spark.rapids.RmmRapidsRetryIterator$.drainSingleWithVerification(RmmRapidsRetryIterator.scala:291)
	at com.nvidia.spark.rapids.RmmRapidsRetryIterator$.withRetryNoSplit(RmmRapidsRetryIterator.scala:164)
	at com.nvidia.spark.rapids.GpuCompressionAwareCoalesceIterator.concatAllAndPutOnGPU(GpuCoalesceBatches.scala:861)
	at com.nvidia.spark.rapids.AbstractGpuCoalesceIterator.$anonfun$next$2(GpuCoalesceBatches.scala:641)
	at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30)
	at com.nvidia.spark.rapids.AbstractGpuCoalesceIterator.$anonfun$next$1(GpuCoalesceBatches.scala:625)
	at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30)
	at com.nvidia.spark.rapids.AbstractGpuCoalesceIterator.next(GpuCoalesceBatches.scala:600)
	at com.nvidia.spark.rapids.AbstractGpuCoalesceIterator.next(GpuCoalesceBatches.scala:250)
	at com.nvidia.spark.rapids.GpuSortEachBatchIterator.next(GpuSortExec.scala:165)
	at com.nvidia.spark.rapids.GpuSortEachBatchIterator.next(GpuSortExec.scala:151)
	at com.nvidia.spark.rapids.ColumnarToRowIterator.$anonfun$fetchNextBatch$3(GpuColumnarToRowExec.scala:290)
	at com.nvidia.spark.rapids.Arm$.withResource(Arm.scala:30)
	at com.nvidia.spark.rapids.ColumnarToRowIterator.fetchNextBatch(GpuColumnarToRowExec.scala:287)
	at com.nvidia.spark.rapids.ColumnarToRowIterator.loadNextBatch(GpuColumnarToRowExec.scala:257)
	at com.nvidia.spark.rapids.ColumnarToRowIterator.hasNext(GpuColumnarToRowExec.scala:304)
	at scala.collection.Iterator$$anon$10.hasNext(Iterator.scala:460)
	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$GeneratedIteratorForCodegenStage2.processNext(Unknown Source)
	at org.apache.spark.sql.execution.BufferedRowIterator.hasNext(BufferedRowIterator.java:43)
	at org.apache.spark.sql.execution.WholeStageCodegenExec$$anon$1.hasNext(WholeStageCodegenExec.scala:760)
	at com.nvidia.spark.rapids.RowToColumnarIterator.hasNext(GpuRowToColumnarExec.scala:604)
	at org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase$$anon$1.partNextBatch(GpuShuffleExchangeExecBase.scala:332)
	at org.apache.spark.sql.rapids.execution.GpuShuffleExchangeExecBase$$anon$1.hasNext(GpuShuffleExchangeExecBase.scala:355)
	at org.apache.spark.shuffle.sort.BypassMergeSortShuffleWriter.write(BypassMergeSortShuffleWriter.java:140)
	at org.apache.spark.shuffle.ShuffleWriteProcessor.write(ShuffleWriteProcessor.scala:59)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
	at org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:52)
	at org.apache.spark.scheduler.Task.run(Task.scala:136)
	at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:548)
	at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1504)
	at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:551)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Spark version: 3.3.0
spark-rapids version: 24.10

Spark configuration:
spark.driver.cores 1
spark.driver.memory 10g
spark.executor.cores 8
spark.executor.memory 50g
spark.executor.resource.gpu.amount 1
spark.task.resource.gpu.amount 0.125
spark.task.cpus 1
spark.rapids.memory.pinnedPool.size 2g
spark.rapids.sql.concurrentGpuTasks 2

[mattahrens]

Thanks for your question and providing details about the issue you are encountering. We will investigate the exception and get back to you with next steps.

One question -- in what environment are you running Spark (e.g. YARN)? The CPU out of memory error is happening in off heap memory which is controlled by the spark.executor.memoryOverhead setting.

CC: @revans2 @viadea

[SidWeng]

We use Standalone cluster

[revans2]

Looking at the code I think we have to have a bug of some kind where we are throwing the wrong type of exception CPU OOM vs GPU OOM.

The reason I say this is that

spark-rapids/sql-plugin/src/main/scala/com/nvidia/spark/rapids/GpuCoalesceBatches.scala

Lines 861 to 863 in 00cd422

	withRetryNoSplit(candidates) { attempt =>
	concatBatches(attempt.toArray)
	}

is the retry block that explicitly is saying that it does not support a split and retry. concatBatches, which is inside of the retry block, only operates on GPU data. There should be no way for us to run out of CPU memory in that block, because we should not be allocating CPU memory in that block, unless we are spilling. And if we are spilling we should be able to handle spilling to disk even if there is no memory CPU memory available.

Can you share what type of GPU you are running on? Also are there other processes that might be sharing that same GPU? The reason I ask is that the processing you are doing does not look like it would cause any OOM issues, either CPU or GPU. But I cannot see the entire query so it is a little hard to tell. But what I can see is a coalesce batches followed by a sort, which then goes into a columnar to row, some processing on the CPU and then a row to columnar followed by a shuffle. All of that code has been extensively tested and if we run out of memory the performance may not be great, but we should not crash.

I'll try to reproduce this locally and see what I can come up with.

[SidWeng]

1. We use Standard_NC8as_T4_v3 with Tesla T4 GPU.
2. The cluster is composed by 8 Standard_NC8as_T4_v3 VMs and by our setting it should be one executor per node and we set spark.rapids.sql.concurrentGpuTasks 2. Only the Spark executor would use the GPU.
3. Because the CPU memory usage is never higher than 50% (we set 50GB to the executor memory, it should be enough.) I guess it might related to GPU OOM and try all the tips from [How to handle GPU OOM issues](https://docs.nvidia.com/spark-rapids/user-guide/latest/best-practices.html#how-to-handle-gpu-oom-issues) but still not work.

[SidWeng]

Our query would be like:

t1.withColumn("alignmentEnd", alignmentEndUDF(col("alignmentStart"), col("raw")))
  .withColumn("parts", partsUDF(col("referenceName"), col("alignmentStart"), col("alignmentEnd")))
  .withColumn("order", getKeyUDF(col("referenceName"), col("alignmentStart")))
  .select(col("raw"), col("order"), functions.explode(col("parts")), col("fqn"))
  .withColumn("partId", col("col"))
  .repartition(numPartitions(), col("partId"))
  .sortWithinPartitions(col("partId"), col("order"))
  .drop("col", "order")