Details
-
Bug
-
Status: Resolved
-
Major
-
Resolution: Fixed
-
3.1.2
-
None
-
Linux RHEL7
Spark 3.1.1
Description
When running Spark on YARN, the user-class-path argument to CoarseGrainedExecutorBackend is used to pass a list of user JAR URIs to executor processes. The argument is specified once for each JAR, and the URIs are fully-qualified, so the paths can be quite long. With large user JAR lists (say 1000+), this can result in system-level argument length limits being exceeded, typically manifesting as the error message:
/bin/bash: Argument list too long
A Google search indicates that this is not a theoretical problem and afflicts real users, including ours. This issue was originally observed on Spark 2.3, but has been confirmed to exist in the master branch as well.
Attachments
Issue Links
- links to