[VL] When the regexExpr of the split function is an empty character or a non-ASCII character, fall back to Valina Spark

[mcdull-zhang]

Backend

VL (Velox)

Bug description

empty character

Expression:  split('abc','')
Spark: ["a","b","c",""]
Gluten: Reason: (0 vs. 1) split only supports only single-character pattern

non-ASCII character

Expression:  split('a，b，c','，')
Spark: ["a","b","c"]
Gluten: Reason: (3 vs. 1) split only supports only single-character pattern

Spark version

Spark-3.2.x

Spark configurations

No response

System information

No response

Relevant logs

java.lang.RuntimeException: Exception: VeloxRuntimeError
Error Source: RUNTIME
Error Code: INVALID_STATE
Reason: (0 vs. 1) split only supports only single-character pattern
Retriable: False
Expression: patternString.size() == 1
Context: split(n0_0, :VARCHAR)
Top-Level Context: Same as context.
Function: apply
File: ep/build-velox/build/velox_ep/velox/functions/sparksql/SplitFunctions.cpp

[jackylee-ch]

This is because the current split only supports splitting delemiter, while spark supports regexp. If necessary, I can add a PR to fix this problem, because Velox already supports regexp_split.

[mcdull-zhang]

@jackylee-ch Please fix it.

[gaoyangxiaozhu]

hey @mcdull-zhang i am working on fixing split issue, there is a fix PR in velox facebookincubator/velox#10248, once the PR ready, i will update gluten part.

[gaoyangxiaozhu]

This is because the current split only supports splitting delemiter, while spark supports regexp. If necessary, I can add a PR to fix this problem, because Velox already supports regexp_split.

hey @jackylee-ch regex_split can't be used here since it not support limit parameter, please check my this Pr facebookincubator/velox#10248

[jackylee-ch]

hey @jackylee-ch regex_split can't be used here since it not support limit parameter, please check my this Pr facebookincubator/velox#10248

Great job, wait for your fix.