Shuffle cannot be removed expectly.

[jackylee-ch]

Backend

VL (Velox)

Bug description

When testing tpcds, we found that the disk space utilization rate was increasing, which eventually led to the failure of tpcds due to insufficient disk space. After investigation, we found that this #6022 caused the shuffle to not be cleaned up in time, resulting in the continuous increase of block space.

BTW, we are running with Spark3.5.1 and I haven't test other Spark Versions.

Spark version

None

Spark configurations

No response

System information

No response

Relevant logs

No response

[zhli1142015]

cc @acvictor

[acvictor]

@jackylee-ch thanks for reporting this! Have you already investigated and working on a fix?

[acvictor]

cc @marin-ma

[jackylee-ch]

@jackylee-ch thanks for reporting this! Have you already investigated and working on a fix?

not yet. I have revert it in out branch.

[acvictor]

@jackylee-ch thanks for reporting this! Have you already investigated and working on a fix?

not yet. I have revert it in out branch.

Thanks, I am working on fixing this.