34 Pull requests opened by 28 people

• [SPARK-48936][CONNECT] Makes spark-submit works with Spark connect

  #47434 opened Jul 22, 2024

• [SPARK-44884][Core] Create _SUCCESS marker file for spark write with …

  #47439 opened Jul 22, 2024

• [SPARK-48346][SQL] Support for IF ELSE statements in SQL scripts

  #47442 opened Jul 22, 2024

• [SPARK-48977][SQL] Optimize string searching under UTF8_LCASE collation

  #47444 opened Jul 22, 2024

• [SPARK-48966][SQL] Improve error message with invalid unresolved column reference in UDTF call

  #47447 opened Jul 22, 2024

• [SPARK-47047] TransformWithState integration with statedata source integration - Value State

  #47450 opened Jul 22, 2024

• [SPARK-48762][SQL] Introduce clusterBy DataFrameWriter API for Python

  #47452 opened Jul 22, 2024

• [TEST janino] v3.1.12 VS v3.1.9

  #47455 opened Jul 23, 2024

• [MINOR][DOC] Fix window function docs

  #47456 opened Jul 23, 2024

• [SPARK-48964][SQL][DOCS] Fix the discrepancy between implementation, comment and documentation of option `recursive.fields.max.depth` in ProtoBuf connector

  #47458 opened Jul 23, 2024

• [SPARK-48980][ML] Avoid per-row param read in `LSH/DCT/NGram/PolynomialExpansion`

  #47461 opened Jul 23, 2024

• [WIP][SPARK-48347][SQL] Support for WHILE statement

  #47462 opened Jul 23, 2024

• [SPARK-48985][CONNECT] Connect Compatible Expression Constructors

  #47464 opened Jul 24, 2024

• [SPARK-48997][SS] Implement individual unloads for maintenance thread pool thread failures

  #47475 opened Jul 25, 2024

• [SPARK-48989][SQL] Fix error result of throwing an exception when the `count` parameter of `SUBSTRING_INDEX`(with Codegen) is null or string

  #47481 opened Jul 25, 2024

• [SPARK-49000][SQL] Fix aggregate behaviour with DISTINCT literals

  #47482 opened Jul 25, 2024

• [SPARK-48910][SQL] Use HashSet/HashMap to avoid linear searches in PreprocessTableCreation

  #47484 opened Jul 25, 2024

• [SPARK-49002][SQL] Consistently handle invalid location/path values for all database objects

  #47485 opened Jul 25, 2024

• Mima

  #47487 opened Jul 25, 2024

• [SPARK-49006] Implement purging for OperatorStateMetadataV2 and StateSchemaV3 files

  #47490 opened Jul 25, 2024

• Rebased value state

  #47492 opened Jul 25, 2024

• [SPARK-49008][PYTHON] Use `ParamSpec` to propagate `func` signature in `transform`

  #47493 opened Jul 25, 2024

• [SPARK-49014][BUILD] Bump Apache Avro to 1.12.0

  #47498 opened Jul 26, 2024

• [SPARK-49000][SQL][WIP] Fix "select count(distinct 1) from t" where t is empty table

  #47499 opened Jul 26, 2024

• [SPARK-49015][CORE] Connect Server should respect `spark.log.structuredLogging.enabled`

  #47500 opened Jul 26, 2024

• [SPARK-49017][SQL] Insert statement fails when multiple parameters are being used

  #47501 opened Jul 26, 2024

• [SPARK-49003][SQL] Ilicmarkodb/fix string hash

  #47502 opened Jul 26, 2024

• [SPARK-49018][SQL] Fix approx_count_distinct not working correctly with collation

  #47503 opened Jul 26, 2024

• [WIP] Test removing the `hadoop.security.key.provider.path` configuration with global scope.

  #47504 opened Jul 26, 2024

• [WIP][SPARK-49000][SQL] Fix aggregation for distinct literal

  #47505 opened Jul 26, 2024

• [SPARK-49016][SQL] Queries from raw CSV files are disallowed when the referenced columns only include the internal corrupt record column

  #47506 opened Jul 26, 2024

• [SPARK-48931][SS][FOLLOWUP] Reduce Cloud Store List API cost for state store maintenance task

  #47507 opened Jul 26, 2024

• [SPARK-49031] Implement validation for the TransformWithStateExec operator using OperatorStateMetadataV2

  #47508 opened Jul 26, 2024

• [SPARK-49032][SS] Add schema path in metadata table entry, verify expected version and add operator metadata related test for operator metadata format v2

  #47510 opened Jul 27, 2024

32 Unresolved conversations

Sometimes conversations happen on old items that aren’t yet closed. Here is a list of all the Issues and Pull Requests with unresolved conversations.

• [SPARK-48939][AVRO] Support reading Avro with recursive schema reference

  #47425 commented on Jul 25, 2024 • 23 new comments

• [SPARK-48900] Add `reason` field for all internal calls for job/stage cancellation

  #47374 commented on Jul 26, 2024 • 19 new comments

• [SPARK-48700] [SQL] Mode expression for complex types (all collations)

  #47154 commented on Jul 27, 2024 • 16 new comments

• [SPARK-48344][SQL] SQL API change to support execution of compound statements

  #47403 commented on Jul 26, 2024 • 14 new comments

• [SPARK-48755] State V2 base implementation and ValueState support

  #47133 commented on Jul 26, 2024 • 11 new comments

• [SPARK-48906][SQL] Introduce `SHOW COLLATIONS LIKE ...` syntax to show all collations

  #47364 commented on Jul 23, 2024 • 10 new comments

• [SPARK-48901][SPARK-48916][SS][PYTHON] Introduce clusterBy DataStreamWriter API

  #47376 commented on Jul 25, 2024 • 7 new comments

• [SPARK-48967][SQL] Improve performance and memory footprint of "INSERT INTO ... VALUES" Statements

  #47428 commented on Jul 24, 2024 • 5 new comments

• [SPARK-48911][SQL][TESTS] Improve collation support testing for various expressions

  #47372 commented on Jul 22, 2024 • 5 new comments

• [SPARK-48881][SQL] Some dynamic partitions can be compensated to specific partition values

  #47418 commented on Jul 23, 2024 • 5 new comments

• [SPARK-42307][SQL] Assign name for error _LEGACY_ERROR_TEMP_2232

  #47354 commented on Jul 27, 2024 • 3 new comments

• [SPARK-48628][CORE] Add task peak on/off heap memory metrics

  #47192 commented on Jul 25, 2024 • 3 new comments

• [SPARK-43301][CORE][SHUFFLE] BlockStoreClient getHostLocalDirs RPC supports IOException retry

  #46805 commented on Jul 22, 2024 • 2 new comments

• [SPARK-48400][CORE]Promote `PrometheusServlet` to `DeveloperApi`

  #46716 commented on Jul 22, 2024 • 1 new comment

• [SPARK-48882][SS] Assign names to streaming output mode related error classes

  #47327 commented on Jul 25, 2024 • 1 new comment

• [SC-170296] GROUP BY with MapType nested inside complex type

  #47331 commented on Jul 26, 2024 • 1 new comment

• [SPARK-48936][CONNECT] Makes spark-shell works with Spark connect

  #47402 commented on Jul 27, 2024 • 0 new comments

• [SPARK-48948][SQL] Introduce `SHOW VARIABLES LIKE ... ` SQL syntax to get variables

  #47422 commented on Jul 23, 2024 • 0 new comments

• [TEST][WIP][][SQL] Introduction of error handling mechanism in SQL Scripting

  #47356 commented on Jul 25, 2024 • 0 new comments

• [WIP][SPARK-48353][SQL] Introduction of Error Handling mechanism in SQL Scripting

  #47423 commented on Jul 26, 2024 • 0 new comments

• [SPARK-48978][SQL] Implement ASCII fast path in collation support for UTF8_LCASE

  #47326 commented on Jul 23, 2024 • 0 new comments

• [SPARK-48856][SQL] Use isolated JobArtifactSet for each spark session

  #47281 commented on Jul 23, 2024 • 0 new comments

• [SPARK-48829][BUILD] Upgrade `RoaringBitmap` to 1.2.1

  #47247 commented on Jul 26, 2024 • 0 new comments

• [SPARK-48821][SQL] Support Update in DataFrameWriterV2

  #47233 commented on Jul 26, 2024 • 0 new comments

• [ONLY TEST][HOLD] Upgrade rocksdbjni to 9.4.0

  #47207 commented on Jul 26, 2024 • 0 new comments

• [SPARK-48835] Introduce versoning to jdbc connectors

  #47181 commented on Jul 22, 2024 • 0 new comments

• [SPARK-48725][SQL] Integrate CollationAwareUTF8String.lowerCaseCodePoints into string expressions

  #47132 commented on Jul 24, 2024 • 0 new comments

• [SPARK-48637][CORE] On-demand shuffle migration peer refresh during decommission

  #46995 commented on Jul 24, 2024 • 0 new comments

• [SPARK-48590][BUILD] Upgrade netty to `4.1.112.Final`

  #46945 commented on Jul 22, 2024 • 0 new comments

• [SPARK-47336][SQL][CONNECT] Provide to PySpark a functionality to get estimated size of DataFrame in bytes

  #46368 commented on Jul 27, 2024 • 0 new comments

• [SPARK-47870][SQL] Optimize predicate after push extra predicate through join

  #46085 commented on Jul 27, 2024 • 0 new comments

• [SPARK-43242][CORE] Fix throw 'Unexpected type of BlockId' in shuffle corruption diagnose

  #40921 commented on Jul 22, 2024 • 0 new comments