2024-10-21

Changes in 4.0.0 release since 3.3.0 release preparation was started.
For the complete list, check the full changelog and full PR list.

What's New in Apache Pulsar 4.0? covers topics such as:

• Enhanced Key_Shared subscription implementation
• Secure Docker runtime based on Alpine Linux and Java 21
• Enhanced Quality of Service controls for multi-tenant deployments
• Rate limiting as a foundation for multi-tenant resource management
• Upgrade paths from previous versions
• Client compatibility and Importance of upgrading Java clients to address CVE-2024-47561

PIPs accepted since 3.0.0 release

PIPs accepted in 4.0.0 release

• PIP-327: Support force topic loading for unrecoverable errors (#21752)
• PIP-337: SSL Factory Plugin to customize SSL Context and SSL Engine generation (#22016)
• PIP-347: add role field in consumer's stat (#22564)
• PIP-348: Trigger offload on topic load stage (#22650)
• PIP-349: Add additionalSystemCursorNames ignore list for TTL check (#22651)
• PIP-350: Allow to disable the managedLedgerOffloadDeletionLagInMillis (#22688)
• PIP-351: Additional options for Pulsar-Test client to support KeyStore based TLS (#22694)
• PIP-352: Event time based topic compactor (#22710)
• PIP-353: Improve transaction message visibility for peek-messages (#22746)
• PIP-354: apply topK mechanism to ModularLoadManagerImpl (#22765)
• PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22778)
• PIP-356: Support Geo-Replication starts at earliest position (#22806)
• PIP-357: Correct the conf name in load balance module. (#22823)
• PIP-358: let resource weight work for OverloadShedder, LeastLongTermMessageRate, ModularLoadManagerImpl. (#22889)
• PIP-359: Support custom message listener executor for specific subscription (#22902)
• PIP-360: Add admin API to display Schema metadata (#22913)
• PIP-363: Add callback parameters to the method: org.apache.pulsar.client.impl.SendCallback.sendComplete. (#22940)
• PIP-364: Introduce a new load balance algorithm AvgShedder (#22946)
• PIP-366: Support to specify different config for Configuration and Local Metadata Store (#23033)
• PIP-367: Propose a Contributor Repository for Pulsar (#23061)
• PIP-368: Support lookup based on the lookup properties (#23075)
• PIP-369: Flag based selective unload on changing ns-isolation-policy (#23116)
• PIP-370: configurable remote topic creation in geo-replication (#23124)
• PIP-374: Visibility of messages in receiverQueue for the consumers (#23235)
• PIP-376: Make topic policies service pluggable (#23248)
• PIP-378: Add ServiceUnitStateTableView abstraction (ExtensibleLoadMangerImpl only) (#23300)
• PIP-379: Key_Shared Draining Hashes for Improved Message Ordering (#23309)
  • Documentation: Preserving order of message delivery by key
• PIP-381: Handle large PositionInfo state (#23328)
• PIP-383: Support granting/revoking permissions for multiple topics (#23355)
• PIP-384: ManagedLedger interface decoupling (#23363)

PIPs accepted in 3.3.0 release

• PIP-315: Configurable max delay limit for delayed delivery (#21798)
• PIP-321: Introduce allowed-cluster at the namespace level (#21648)
• PIP-324: Alpine Docker images (#21716)
• PIP-325: Add command to abort transaction (#21731)
• PIP-326: Create a BOM to ease dependency management (#21747)
• PIP-329: Strategy for maintaining the latest tag to Pulsar docker images (#21872)
• PIP-330: getMessagesById gets all messages (#21873)
• PIP-342: Support OpenTelemetry metrics in Pulsar client (#22178)
• PIP-343: Use picocli instead of jcommander (#22181)
• PIP-344: Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) (#22182)
• PIP-335: Oxia metadata plugin (#22009)
• PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22071)
• PIP-282: Change definition of the recently joined consumers position (#20776)

PIPs accepted in 3.2.0 release

• PIP-323: Complete Backlog Quota Telemetry (#21709)
• PIP-322: Pulsar Rate Limiting Refactoring (#21680)
• PIP-320: OpenTelemetry Scaffolding (#21635)
• PIP-318: Don't retain null-key messages during topic compaction (#21541)
• PIP-315: Configurable max delay limit for delayed del...

2024-10-04

Library updates

• [fix][sec] Upgrade Avro to 1.11.4 to address CVE-2024-47561 (#23394)
• [fix][sec] Upgrade vertx to 4.5.10 to address CVE-2024-8391 (#23338)
• [fix][sec][branch-3.3] Upgrade protobuf-java to 3.25.5 (#23356) (#23358)
• [improve][misc] Upgrade Netty to 4.1.113 and netty-tcnative to 2.0.66 (#23255)
• [fix] Bump commons-io:commons-io from 2.8.0 to 2.14.0 (#23393)
• [fix] Bump io.grpc from 1.56.0 to 1.56.1 (#23276)
• [fix][metadata] Upgrade Oxia to 0.3.2 (#23140)
• [feat][meta] Bump oxia java version from 0.3.2 to 0.4.5 (#23277)
• [improve] Upgrade Pulsar Python client in docker image to 3.5.0 (#23377)
• [improve] Install openssl in the docker image to fix compatibility with Apache Pulsar Helm chart (#23362)

Broker

• [fix][broker] Cancel possible pending replay read in cancelPendingRead (#23384)
• [fix][broker] Execute the pending callbacks in order before ready for incoming requests (#23266)
• [fix][broker] Fail fast if the extensible load manager failed to start (#23297)
• [fix][broker] Fix 'Disabled replicated subscriptions controller' logic and logging (#23142)
• [fix][broker] Fix authenticate order in AuthenticationProviderList (#23111)
• [fix][broker] Fix brokers still retry start replication after closed the topic (#23237)
• [fix][broker] Fix compatibility issues for PIP-344 (#23136)
• [fix][broker] Fix incomplete NAR file extraction which prevents broker from starting (#23274)
• [fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector (#23327)
• [fix][broker] Fix shadow topics cannot be consumed when the entry is not cached (#23147)
• [fix][broker] Fix system topic can not be loaded up if it contains data offloaded (#23279)
• [fix][broker] Fix the broker registery cannot recover from the metadata node deletion (#23359)
• [fix][broker] Fix the bug that elected leader thinks it's a follower (#23138)
• [fix][broker] Handle the case when getOwnedServiceUnits fails gracefully (#23119)
• [fix][broker] Internal reader of __change_events can not started after metadata store session rebuilt (#23018)
• [fix][broker] Let Pending ack handler can retry to init when encounters a metadata store error (#23153)
• [fix][broker] Skip reading entries from closed cursor. (#22751)
• [fix][broker] Topic can never be loaded up due to broker maintains a failed topic creation future (#23184)
• [fix][broker] fix exception may hidden and result in stuck when topic loading (#23102)
• [fix][broker] fix pulsar-admin topics stats-internal caused a BK client thread a deadlock (#23258)
• [fix][broker] fix replicated namespaces filter in filterAndUnloadMatchedNamespaceAsync (#23100)
• [fix][broker] type cast on exceptions in exceptionally can lead to lost calls (#23117)
• [fix][broker]A failed consumer/producer future in ServerCnx can never be removed (#23123)
• [improve][broker] Add msgInReplay subscription stat and metric to improve Key_Shared observability (#23224)
• [improve][broker] Add retry for start service unit state channel (ExtensibleLoadManagerImpl only) (#23230)
• [improve][broker] Avoid subscription fenced error with consumer.seek whenever possible (#23163)
• [improve][broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only) (#23144)
• [improve][broker] Improve pulsar_topic_load_failed metric to record correct failed time (#23199)
• [improve][broker] Optimize high CPU usage when consuming from topics with ongoing txn (#23189)
• [improve][broker] Optimize message payload traffic for ShadowReplicator (#23236)
• [improve][broker] Optimize performance for checking max topics when the topic is a system topic (#23185)
• [improve][broker] Optimize the performance of individual acknowledgments (#23072)
• [improve][broker] Register the broker to metadata store without version id compare (#23298)
• [improve][broker] Should notify bundle ownership listener onLoad event when ServiceUnitState start (ExtensibleLoadManagerImpl only) (#23152)
• [improve][broker] Support customized shadow managed ledger implementation (#23179)
• [improve][broker] Support to specify auth-plugin, auth-parameters and tls-enable arguments when init cluster metadata (#23087)
• [improve][broker] Reuse method getAvailableBrokersAsync (#23099)
• [improve][admin] PIP-369 Introduce unload flag in ns-isolation-policy set call (#23120)
• [fix] StatsOutputStream: add string write function (#23227)

Client

• [fix] DLQ to handle bytes key properly (#23172)
• [fix][client] Copy orderingKey to retry letter topic and DLQ messages and fix bug in copying (#23182)
• [fix][client] Create the retry producer async (#23157)
• [fix][client] Fix for early hit beforeConsume for MultiTopicConsumer (#23141)
• [fix][client] Fix timeout handling in Pulsar Admin client (#23128)
• [fix][client] TransactionCoordinatorClient support retry (#23081)
• [fix][client] the nullValue in msgMetadata should be true by default (#22372)
• [improve][client] Add maxConnectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder (#22541)
• [improve][client] Don't print info logs for each schema loaded by client (#23206)
• [improve][client] Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover (#23129)

Pulsar IO and Pulsar Functions

• [fix][io] Upgrade mssql server docker tag in DebeziumMsSqlContainer (#23318)
• [improve][fn] Add support for overriding additionalJavaRuntimeArguments with PF_additionalJavaRuntimeArguments env (#23130)

Others

• [fix][log] Do not print error log if tenant/namespace does not exist when calling get topic metadata (#23291)
• [fix][log] Do not print warn log when concurrently publishing and switching ledgers (#23209)
• [feat] Add scripts for updating BK RocksDB ini files (#23178)
• [fix][meta] Oxia metadta store: Convert error to MetadataStoreException if operation failed (#23154)
• [fix][misc] Log Conscrypt security provider initialization warnings at debug level (#23364)
• [improve][misc] Improve AES-GCM cipher performance (#23122)
• [improve][misc] Optimize TLS performance by omitting extra buffer copies (#23115)
• [improve][pip] PIP-366: Support to specify different config for Configuration and Local Metadata Store (#23041)
• [improve][proxy] Reuse authentication instance in pulsar-proxy (#23113)

Tests & CI

• [fix][test] Fix flaky SubscriptionSeekTest.testSeekIsByReceive (#23170)
• [fix][test] Fix flaky UnloadSubscriptionTest.testMultiConsumer ([#23243](#23243...

2024-10-04

Library updates

• [fix][sec] Upgrade Avro to 1.11.4 to address CVE-2024-47561 (#23394)
• [fix][sec] Upgrade vertx to 4.5.10 to address CVE-2024-8391 (#23338)
• [fix][sec][branch-3.0] Upgrade protobuf-java to 3.25.5 (#23356) (#23357)
• [fix] Bump commons-io:commons-io from 2.8.0 to 2.14.0 (#23393)
• [improve][misc] Upgrade Netty to 4.1.113 and netty-tcnative to 2.0.66 (#23255)
• [improve][misc] Upgrade jersey to 2.41 (#22290)
• [improve] Upgrade Pulsar Python client in docker image to 3.5.0 (#23377)

Broker

• [fix][broker] Cancel possible pending replay read in cancelPendingRead (#23384)
• [fix][broker] Execute the pending callbacks in order before ready for incoming requests (#23266)
• [fix][broker] Fix 'Disabled replicated subscriptions controller' logic and logging (#23142)
• [fix][broker] Fix authenticate order in AuthenticationProviderList (#23111)
• [fix][broker] Fix brokers still retry start replication after closed the topic (#23237)
• [fix][broker] Fix compatibility issues for PIP-344 (#23136)
• [fix][broker] Fix incomplete NAR file extraction which prevents broker from starting (#23274)
• [fix][broker] Fix out-of-order issues with ConsistentHashingStickyKeyConsumerSelector (#23327)
• [fix][broker] Fix shadow topics cannot be consumed when the entry is not cached (#23147)
• [fix][broker] Fix system topic can not be loaded up if it contains data offloaded (#23279)
• [fix][broker] Fix the bug that elected leader thinks it's a follower (#23138)
• [fix][broker] Handle the case when getOwnedServiceUnits fails gracefully (#23119)
• [fix][broker] Internal reader of __change_events can not started after metadata store session rebuilt (#23018)
• [fix][broker] Let Pending ack handler can retry to init when encounters a metadata store error (#23153)
• [fix][broker] Skip reading entries from closed cursor. (#22751)
• [fix][broker] Topic can never be loaded up due to broker maintains a failed topic creation future (#23184)
• [fix][broker] fix exception may hidden and result in stuck when topic loading (#23102)
• [fix][broker] fix pulsar-admin topics stats-internal caused a BK client thread a deadlock (#23258)
• [fix][broker] fix replicated namespaces filter in filterAndUnloadMatchedNamespaceAsync (#23100)
• [fix][broker] fix the log format error in the AdminResource (#23278)
• [fix][broker] type cast on exceptions in exceptionally can lead to lost calls (#23117)
• [fix][broker]A failed consumer/producer future in ServerCnx can never be removed (#23123)
• [fix][broker][branch-3.0] Fail fast if the extensible load manager failed to start (#23297) (#23302)
• [fix][broker] StatsOutputStream: add string write function (#23227)
• [improve][broker] Add msgInReplay subscription stat and metric to improve Key_Shared observability (#23224)
• [improve][broker] Add retry for start service unit state channel (ExtensibleLoadManagerImpl only) (#23230)
• [improve][broker] Avoid subscription fenced error with consumer.seek whenever possible (#23163)
• [improve][broker] Explicitly close LB internal topics when playing a follower (ExtensibleLoadManagerImpl only) (#23144)
• [improve][broker] Improve pulsar_topic_load_failed metric to record correct failed time (#23199)
• [improve][broker] Optimize high CPU usage when consuming from topics with ongoing txn (#23189)
• [improve][broker] Optimize performance for checking max topics when the topic is a system topic (#23185)
• [improve][broker] Optimize the performance of individual acknowledgments (#23072)
• [improve][broker] Part 2 of PIP-370: add metrics "pulsar_replication_disconnected_count" (#23213)
• [improve][broker] Phase 1 of PIP-370 support disable create topics on remote cluster through replication (#23169)
• [improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)
• [improve][broker] Should notify bundle ownership listener onLoad event when ServiceUnitState start (ExtensibleLoadManagerImpl only) (#23152)
• [improve][broker] Support customized shadow managed ledger implementation (#23179)
• [improve][broker] Support to specify auth-plugin, auth-parameters and tls-enable arguments when init cluster metadata (#23087) (#23126)
• [improve][broker] Reuse method getAvailableBrokersAsync (#23099)
• [improve][admin] PIP-369 Introduce unload flag in ns-isolation-policy set call (#23120)

Client

• [fix] DLQ to handle bytes key properly (#23172)
• [fix][client] Copy orderingKey to retry letter topic and DLQ messages and fix bug in copying (#23182)
• [fix][client] Create the retry producer async (#23157)
• [fix][client] Fix for early hit beforeConsume for MultiTopicConsumer (#23141)
• [fix][client] Fix timeout handling in Pulsar Admin client (#23128)
• [fix][client] TransactionCoordinatorClient support retry (#23081)
• [fix][client] fix negative message re-delivery twice issue (#20750)
• [fix][client] the nullValue in msgMetadata should be true by default (#22372)
• [improve][client] Add maxConnectionsPerHost and connectionMaxIdleSeconds to PulsarAdminBuilder (#22541)
• [improve][client] Don't print info logs for each schema loaded by client (#23206)
• [improve][client] Swallow Conscrypt ClassNotFoundException (#20371)
• [improve][client]Add new ServiceUrlProvider implementation: SameAuthParamsAutoClusterFailover (#23129)
• [improve][client]PIP-359:Support custom message listener executor for specific subscription (#22861)

Pulsar IO and Pulsar Functions

• [fix][fn] Make python install dependencies from requirements.txt (#20174)
• [fix][io] Upgrade mssql server docker tag in DebeziumMsSqlContainer (#23318)
• [improve][fn] Add support for overriding additionalJavaRuntimeArguments with PF_additionalJavaRuntimeArguments env (#23130)
• [fix][sql][branch-3.0] Fix UUID convert failed for JSON schema (#23292)

Others

• [feat] Add scripts for updating BK RocksDB ini files (#23178)
• [fix][log] Do not print error log if tenant/namespace does not exist when calling get topic metadata (#23291)
• [fix][log] Do not print warn log when concurrently publishing and switching ledgers (#23209)
• [fix][misc] Log Conscrypt security provider initialization warnings at debug level (#23364)
• [improve][misc] Improve AES-GCM cipher performance (#23122)
• [improve][misc] Optimize TLS performance by omitting extra buffer copies (#23115)
• [improve][proxy] Reuse authentication instance in pulsar-proxy (#23113)

Tests & CI...

2024-08-01

Broker

• [feat][broker] PIP-264: Add OpenTelemetry consumer metrics (#22693)
• [feat][broker] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22961)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Remove blocking calls from Subscription.getStats (#23088)
• [fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
• [fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
• [fix][broker] The topic might reference a closed ledger (#22860)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] disable loadBalancerDirectMemoryResourceWeight by default (#22821)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] maintain last active info in memory only. (#22794)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.3] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23085)
• [fix][misc] Disable JFR based telemetry collection since it's not used (#22869)
• [fix][misc] Topic name from persistence name should decode local name (#22879)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [improve] Validate range of argument before long -> int conversion (#22830)
• [improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)
• [improve][broker] Add subscription prefix for internal reader (#23044)
• [improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
• [improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
• [improve][broker] Do not call cursor.isCursorDataFullyPersistable if disabled dispatcherPauseOnAckStatePersistentEnabled (#22729)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] PIP-355: Enhancing Broker-Level Metrics for Pulsar (#22779)
• [improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
• [improve][broker] Reduce number of OpenTelemetry consumer attributes (#22837)
• [improve][broker] Reduce the CPU pressure from the transaction buffer in rolling restarts (#23062)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Reuse topic OpenTelemetry attributes (#22876)
• [improve][broker] Trigger offload on topic load (#22652)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] high CPU usage caused by list topics under namespace (#23049)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
• [fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted ([#22854](#22...

2024-08-01

Broker

• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Support advertised listeners when gracefully transferring topics (ExtensibleLoadManagerImpl only) (#22862)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] The topic might reference a closed ledger (#22860)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] maintain last active info in memory only. (#22794)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.2] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23084)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [fix][misc] Topic name from persistence name should decode local name (#22879)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
• [improve][broker] Close protocol handlers before unloading namespace bundles (#22728)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] Include runtime dependencies in server distribution (#22001)
• [improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Trigger offload on topic load (#22652)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix orphan consumer when reconnection and closing are concurrency executing (#22958)
• [fix][client] Fix pattern consumer create crash if a part of partitions of a topic have been deleted (#22854)
• [fix][client] Fix resource leak in Pulsar Client since HttpLookupService doesn't get closed (#22858)
• [improve][client] improve the class GetTopicsResult (#22766)

Pulsar IO and Pulsar Functions

• [fix][fn] Enable optimized Netty direct byte buffer support for Pulsar Function runtimes (#22910)
• [fix][fn] Support compression type and crypto config for all producers in Functions and Connectors (#22950)
• [improve][fn] Make producer cache bounded and expiring in Functions/Connectors (#22945)

Others

• [fix][cli] Fix Pulsar standalone "--wipe-data" (#22885)
• [fix][cli] Fix Pulsar standalone shutdown - bkCluster wasn't closed (#22868)
• [fix][cli] Fix expiration of tokens created with "pulsar tokens create" (#22815)
• [fix][cli] Fix the pulsar-daemon parameter passthrough syntax (#22905)
• [fix][meta] Check if metadata store is closed in RocksdbMetadataStore (#22852)
• [fix][offload] Break the fillbuffer loop when met EOF (#22722)

Library updates

• [fix][sec] Upgrade Bouncycastle libraries to address CVEs (#22826)
• [improve] Upgrade IPAddress to 5.5.0 (#22886)
• [improve] Upgrade Jetcd to 0.7.7 and VertX to 4.5.8 (#22835)
• [cleanup][misc] Remove classifier from netty-transport-native-unix-common dependency (#22951)
• [fix][misc] Remove RoaringBitmap dependency from pulsar-common (#23008)
• [fix][misc] Rename netty native libraries in pulsar-client-admin-shaded (#22954)
• [improve][misc] Replace rename-netty-native-libs.sh script with renaming with maven-shade-plugin (#22957)
• [improve][misc] Upgrade to Netty 4.1.111.Final and switch to use grpc-netty-shaded (#22892)
• [improve][misc][branch-3.2] Upgrade to Bookkeeper 4.16.6 (#22963)

Tests...

2024-08-01

Broker

• [feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry (#21816) (#22740)
• [feat][broker][branch-3.0] PIP-321 Introduce allowed-cluster at the namespace level (#22378) (#22960)
• [fix] Make operations on individualDeletedMessages in lock scope (#22966)
• [fix] Remove blocking calls from BookieRackAffinityMapping (#22846)
• [improve] Refactored BK ClientFactory to return futures (#22853)
• [fix][broker] Asynchronously return brokerRegistry.lookupAsync when checking if broker is active(ExtensibleLoadManagerImpl only) (#22899)
• [fix][broker] Can't connecte to non-persist topic when enable broker client tls (#22991)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Check the markDeletePosition and calculate the backlog (#22947)
• [fix][broker] Ensure that PulsarService is ready for serving incoming requests (#22977)
• [fix][broker] EntryFilters fix NoClassDefFoundError due to closed classloader (#22767)
• [fix][broker] Fix MessageDeduplication replay timeout cause topic loading stuck (#23004)
• [fix][broker] Fix NPE after publishing a tombstone to the service unit channel (#22859)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix broker OOM when upload a large package. (#22989)
• [fix][broker] Fix cursor should use latest ledger config (#22644)
• [fix][broker] Fix geo-replication admin client url (#22584)
• [fix][broker] Fix stuck when enable topic level replication and build remote admin fails (#23028)
• [fix][broker] Fix topic status for oldestBacklogMessageAgeSeconds continuously increases even when there is no backlog. (#22907)
• [fix][broker] Fix updatePartitionedTopic when replication at ns level and topic policy is set (#22971)
• [fix][broker] Fix wrong logic of method TopicName.getPartition(int index) (#19841)
• [fix][broker] Handle BucketDelayedDeliveryTracker recover failed (#22735)
• [fix][broker] Immediately tombstone Deleted and Free state bundles (#22743)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] Messages lost on the remote cluster when using topic level replication (#22890)
• [fix][broker] PulsarStandalone started with error if --stream-storage-port is not 4181 (#22993)
• [fix][broker] Remove blocking calls from Subscription.getStats (#23088)
• [fix][broker] Replication stuck when partitions count between two clusters is not the same (#22983)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] Update init and shutdown time and other minor logic (ExtensibleLoadManagerImpl only) (#22930)
• [fix][broker] broker log a full thread dump when a deadlock is detected in healthcheck every time (#22916)
• [fix][broker] fix deadlock when disable topic level Geo-Replication (#22738)
• [fix][broker] fix topic partitions was expanded even if disabled topic level replication (#22769)
• [fix][broker] response not-found error if topic does not exist when calling getPartitionedTopicMetadata (#22838)
• [fix][broker]Fix lookupService.getTopicsUnderNamespace can not work with a quote pattern (#23014)
• [fix][broker][branch-3.0] Do not try to clean owned bundles from inactive source brokers (ExtensibleLoadManagerImpl only) (#23064) (#23077)
• [fix][broker][branch-3.0] The topic might reference a closed ledger (#22860) (#22900)
• [improve][broker] Check max producers/consumers limitation first before other ops to save resources (#23074)
• [improve][broker] Clear thread local BrokerEntryMetadata instance before reuse (#22752)
• [improve][broker] Follow up #4196 use PulsarByteBufAllocator handle OOM (#20837)
• [improve][broker] GetPartitionMetadata fail also can produce messages (#23050)
• [improve][broker] Improve CPU resources usege of TopicName Cache (#23052)
• [improve][broker] Improve efficiency of checking message deletion (#20490)
• [improve][broker] Improve exception for topic does not have schema to check (#22974)
• [improve][broker] Include runtime dependencies in server distribution (#22001)
• [improve][broker] Optimize PersistentTopic.getLastDispatchablePosition (#22707)
• [improve][broker] PIP-356 Support Geo-Replication starts at earliest position (#22856)
• [improve][broker] Remove ClassLoaderSwitcher to avoid objects allocations and consistent the codestyle (#22796)
• [improve][broker] Use RoaringBitmap in tracking individual acks to reduce memory usage (#23006)
• [improve][broker] [break change] Do not create partitioned DLQ/Retry topic automatically (#22705)
• [improve][broker] avoid creating new objects when intercepting (#22790)
• [improve][broker] checkTopicExists supports checking partitioned topic without index (#21701)
• [improve][broker] do not grant permission for each partition to reduce unnecessary zk metadata (#18222)
• [improve][broker] high CPU usage caused by list topics under namespace (#23049)
• [improve][broker] make system topic distribute evenly. (#22953)
• [improve][broker]Ensure namespace deletion doesn't fail (#22627)
• [improve][broker][branch-3.0] PIP-364: Introduce a new load balance algorithm AvgShedder (#23053)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [fix][ml] Add entry fail due to race condition about add entry failed/timeout and switch ledger (#22221)
• [fix][ml]: subscription props could be lost in case of missing ledger during recovery (#22637)
• [fix][ml] Fix race conditions in RangeCache (#22789)
• [improve][ml] RangeCache refactoring: test race conditions and prevent endless loops (#22814)
• [improve][ml] RangeCache refactoring follow-up: use StampedLock instead of synchronized (#22818)
• [fix][admin] Clearly define REST API on Open API (#22783)
• [fix][admin] Clearly define REST API on Open API for Namesaces@v2 (#22775)
• [fix][admin] Clearly define REST API on Open API for Topics (#22782)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix half deletion when attempt to topic with a incorrect API (#23002)
• [fix][admin][part-1]Clearly define REST API on Open API (#22774)
• [improve][admin] Check if the topic existed before the permission operations (#22742)

Client

• [fix][client] Fix negative acknowledgement by messageId (#23060)
• [fix][client] Fix o...

PIP

• PIP-315: Configurable max delay limit for delayed delivery (#21798)
• PIP-321 Introduce allowed-cluster at the namespace level (#21648)
• PIP-324: Alpine Docker images (#21716)
• PIP-325: Add command to abort transaction (#21731)
• PIP-326: Create a BOM to ease dependency management (#21747)
• PIP-329: Strategy for maintaining the latest tag to Pulsar docker images (#21872)
• PIP-330: getMessagesById gets all messages (#21873)
• PIP 342: Support OpenTelemetry metrics in Pulsar client (#22178)
• PIP-343: Use picocli instead of jcommander (#22181)
• PIP-344 Correct the behavior of the public API pulsarClient.getPartitionsForTopic(topicName) (#22182)
• PIP-335: Oxia metadata plugin (#22009)
• PIP-339: Introducing the --log-topic Option for Pulsar Sinks and Sources (#22071)

Broker

• [admin][broker] Fix force delete subscription not working (#22423)
• [cleanup][admin] Remove unused methods in PersistentTopicsBase (#22424)
• [cleanup][broker] Remove unused NamespaceBundleFactory parameter when creating OwnershipCache (#22482)
• [cleanup][broker]∂ fix doc for TransactionBuffer#isTxnAborted (#21956)
• [cleanup][broker] fix return value of TransactionTimeoutTrackerImpl#addTransaction (#22020)
• [cleanup][broker] remove useless code comment (#22459)
• [cleanup][meta] Remove com.beust.jcommander.internal import (#22294)
• [cleanup][ml] ManagedCursor clean up. (#22246)
• [cleanup][test] remove useless TestAuthorizationProvider2 (#22595)
• [feat][admin] Enable Gzip Compression by Default in Admin Client (#22464)
• [feat][admin] PIP-330: getMessagesById gets all messages (#21918)
• [feat][broker] Implementation of PIP-323: Complete Backlog Quota Telemetry (#21816)
• [feat][broker] PIP-264: Add Java runtime metrics (#22616)
• [feat][broker] PIP-264: Add topic messaging metrics (#22467)
• [feat][misc] Add Pulsar BOM (Bill of Materials) (#21871)
• [feat][misc] PIP-264: Implement topic lookup metrics using OpenTelemetry (#22058)
• [feat][misc] PIP-320: Add OpenTelemetry scaffolding (#22010)
• [fix] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [fix] Fixed implicit conversions from long -> int (#22055)
• [fix] Include swagger annotations in shaded client lib (#22570)
• [fix] Restored method as deprecated in AbstractMetadataStore (#21950)
• [fix] Test was leaving client instance to null (#22631)
• [fix][admin] Clearly define REST API on Open API (#22783)
• [fix][admin] Clearly define REST API on Open API for Namesaces@v2 (#22775)
• [fix][admin] Clearly define REST API on Open API for Topics (#22782)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix deprecated check (#22653)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][admin][part-1]Clearly define REST API on Open API (#22774)
• [fix][broker] Add the missed opentelemetry-sdk-testing dependency to tests of pulsar-broker-auth-sasl (#22665)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Avoid consumers receiving acknowledged messages from compacted topic after reconnection (#21187)
• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Check the broker is available for the SLA monitor bundle when the ExtensibleLoadManager is enabled (#22485)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Continue closing even when executor is shut down (#22599)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Delete compacted ledger when topic is deleted (#21745)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Do not migrate internal topics during Blue-Green Migration when ExtensibleLoadBalancer is used (#22478)
• [fix][broker] Enabling batch causes negative unackedMessages due to ack and delivery concurrency (#22090)
• [fix][broker] Expire messages according to ledger close time to avoid client clock skew (#21940)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix PersistentSubscription duplicate implementation interface Subscription (#22359)
• [fix][broker] Fix Replicated Topic unload bug when ExtensibleLoadManager is enabled (#22496)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix String wrong format (#21829)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix break change: could not subscribe partitioned topic with a suffix-matched regexp due to a mistake of PIP-145 (#21885)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix broker not starting when both transactions and the Extensible Load Manager are enabled (#22139)
• [fix][broker] Fix can not subscribe partitioned topic with a suffix-matched regexp (#22025)
• [fix][broker] Fix compaction/replication data loss when expire messages (#21865)
• [fix][broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented (#22684)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix deadlock while skip non-recoverable ledgers. (#21915)
• [fix][broker] Fix getMessageById throws 500 (#21919)
• [fix][broker] Fix hash collision when using a consumer name that ends with a number (#22053)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix issue with GetMessageIdByTimestamp can't find match messageId from compacted ledger (#21600)
• [fix][broker] Fix leader broker cannot be determined when the advertised address and advertised listeners are configured (#21894)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix potential NPE when disabling the liveness check (#21840)
• [fix][broker] Fix reader stuck when read from compacted topic with read compact mode disable (#21969)
• [fix][broker] Fix returns wrong webServiceUrl when both webServicePort and webServicePortTls are set (#21842)
• [fix][broker] Fix schema deletion error when deleting a partitioned topic with many partitions and schema (#21977)
• [fix][broker] Fix thread safety of loadSheddingTask and loadResourceQuotaTask fields (#22660)
• [fix][broker] Fix typo (#21843)
• [fix][broker] Fix typo in CompactedTopicImpl. (#22235)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] Fix typos in PersistentTopic class (#22364)
• [fix][broker] Fix typos in UniformLoadShedder class (#21907)
• [fix][broker] Fix write all compacted out entry into compacted topic (#21917)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] Make ExtensibleLoadManagerImpl.getOwnedServiceUnits async (#22727)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Replication stopped due to unload topic failed (#21947)
• [fix][broker] Restore the broker id to match the format used in existing Pulsar releases (#21937)
• [fix][broker] Return getOwnerAsync without waiting on source broker upon Assigning and Releasing and handle role change during role init (#22112)
• [fix][broker] Sanitize values before logging in apply-config-from-env.py script (#22044)
• [fix][broker] Set ServiceUnitStateChannel topic compaction threshold explicitly, improve getOwnerAsync, and fix other bugs (#22064)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379)
• [fix][broker] Subscription stuck due to called Admin API analyzeSubscriptionBacklog (#22019)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Support lookup options for extensible load manager (#22487)
• [fix][broker] Support running docker container with gid != 0 (#22081)
• [fix][broker] Unify topic-level policies enable judgment conditions (#19501)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTop...

Broker

• [fix][admin] Fix force delete subscription not working (#22423)
• [fix][broker] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [fix][admin] Fix can't delete tenant for v1 (#22550)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Do not migrate internal topics during Blue-Green Migration when ExtensibleLoadBalancer is used (#22478)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22407)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321) (#22418)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTopic<0 (#22397)
• [fix][broker] avoid offload system topic (#22497)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker][admin] Fix cannot update properties on NonDurable subscription. (#22411)
• [improve][broker] Make the config metricsBufferResponse description more effective (#22490)
• [improve][meta] Retry re-validating ResourceLock with backoff after errors (#22617)
• [improve][admin] Check if the topic existed before the permission operations (#22547)
• [improve][broker] Add additionalSystemCursorNames ignore list for TTL check (#22614)
• [improve][broker] Add topic name to emitted error messages. (#22506)
• [improve][broker] Avoid repeated Read-and-discard when using Key_Shared mode (#22245)
• [improve][broker] Create partitioned topics automatically when enable topic level replication (#22537)
• [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
• [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it (#22370)
• [improve][broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result (#22521)
• [improve][broker] Propagate cause exception in TopicBusyException when applicable (#22596)
• [improve][broker] Repeat the handleMetadataChanges callback when configurationMetadataStore equals localMetadataStore (#22519)
• [improve][broker] Servlet support response compression (#21667)
• [improve][broker] Support X-Forwarded-For and HA Proxy Protocol for resolving original client IP of http/https requests (#22524)
• [improve][broker] backlog quota exceed limit log replaced with debug (#22488)
• [fix][ml] Fix NPE of getValidPositionAfterSkippedEntries when recovering a terminated managed ledger (#22552)
• [fix][ml] Mark delete stuck due to switching cursor ledger fails (#22662)
• [fix][ml] No rollover inactive ledgers when metadata service invalid (#22284)
• [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache (#22679)
• [fix][offload] Increase file upload limit from 2048MiB to 4096MiB for GCP/GCS offloading (#22554)
• [fix][storage] ReadonlyManagedLedger initialization does not fill in the properties (#22630)
• [fix][txn] Handle exceptions in the transaction pending ack init (#21274)
• [improve][meta] Log a warning when ZK batch fails with connectionloss (#22566)
• [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
• [improve][offload] Replace usage of shaded class in OffsetsCache (#22683)

Client

• [fix][client] Include swagger annotations in shaded client lib (#22570)
• [fix][client] Fix Consumer should return configured batch receive max messages (#22619)
• [fix][client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry (#22639)
• [fix][client] Fix client side memory leak when call MessageImpl.create and fix imprecise client-side metrics: pendingMessagesUpDownCounter, pendingBytesUpDownCounter, latencyHistogram (#22393)

Pulsar IO and Pulsar Functions

• [fix][fn] make sure the classloader for ContextImpl is functionClassLoader in different runtimes (#22501)
• [fix][io] CompressionEnabled didn't work on elasticsearch sink (#22565)
• [fix][io] Fix es index creation (#22654)
• [fix][io] Kafka Source connector maybe stuck (#22511)
• [improve][io]: Add ...

Broker

• [fix][broker] Fix force delete subscription not working (#22423)
• [fix][broker] Avoid being stuck when closing the broker with extensible load manager (#22573)
• [fix][broker] Create new ledger after the current ledger is closed (#22034)
• [fix][broker] Disable system topic message deduplication (#22582)
• [fix][broker] Fix BufferOverflowException and EOFException bugs in /metrics gzip compression (#22576)
• [fix][broker] Fix NPE causing dispatching to stop when using Key_Shared mode and allowOutOfOrderDelivery=true (#22533)
• [fix][broker] Fix ProducerBusy issue due to incorrect userCreatedProducerCount on non-persistent topic (#22685)
• [fix][broker] Fix a deadlock in SystemTopicBasedTopicPoliciesService during NamespaceEventsSystemTopicFactory init (#22528)
• [fix][broker] Fix broken topic policy implementation compatibility with old pulsar version (#22535)
• [fix][broker] Fix configurationMetadataSyncEventTopic is marked supporting dynamic setting, but not implemented (#22684)
• [fix][broker] Fix consumer stops receiving messages when with large backlogs processing (#22454)
• [fix][broker] Fix invalid condition in logging exceptions (#22412)
• [fix][broker] Fix message drop record in producer stat (#22458)
• [fix][broker] Fix metrics pulsar_topic_load_failed_count is 0 when load non-persistent topic fails and fix the flaky test testBrokerStatsTopicLoadFailed (#22580)
• [fix][broker] Fix nothing changed after removing dynamic configs (#22673)
• [fix][broker] Fix typos in Consumer class (#22532)
• [fix][broker] Fix typos lister -> listener (#21068)
• [fix][broker] One topic can be closed multiple times concurrently (#17524)
• [fix][broker] Optimize /metrics, fix unbounded request queue issue and fix race conditions in metricsBufferResponse mode (#22494)
• [fix][broker] Part-1: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21946)
• [fix][broker] Part-2: Replicator can not created successfully due to an orphan replicator in the previous topic owner (#21948)
• [fix][broker] Prevent long deduplication cursor backlog so that topic loading wouldn't timeout (#22479)
• [fix][broker] Reader stuck after call hasMessageAvailable when enable replicateSubscriptionState (#22572)
• [fix][broker] Skip topic.close during unloading if the topic future fails with ownership check, and fix isBundleOwnedByAnyBroker to use ns.checkOwnershipPresentAsync for ExtensibleLoadBalancer (#22379) (#22403)
• [fix][broker] Support OIDC providers with JWK without alg field set in keys (#22421)
• [fix][broker] Update TransferShedder underloaded broker check to consider max loaded broker's msgThroughputEMA and update IsExtensibleLoadBalancerImpl check (#22321) (#22416)
• [fix][broker] Update topic partition failed when config maxNumPartitionsPerPartitionedTopic<0 (#22397)
• [fix][broker] avoid offload system topic (#22497)
• [fix][broker] fix replicated subscriptions for transactional messages (#22452)
• [fix][broker] rename to changeMaxReadPositionCount (#22656)
• [fix][broker] usedLocallySinceLastReport should always be reset (#22672)
• [fix][broker][admin] Fix cannot update properties on NonDurable subscription. (#22411)
• [fix][broker] Fix Reader can be stuck from transaction aborted messages. (#22610)
• [improve][broker] Make the config metricsBufferResponse description more effective (#22490)
• [improve][broker] Retry re-validating ResourceLock with backoff after errors (#22617)
• [improve][broker] Add topic_load_failed metric (#19236)
• [improve][broker] Add topic name to emitted error messages. (#22506)
• [improve][broker] Avoid repeated Read-and-discard when using Key_Shared mode (#22245)
• [improve][broker] Create partitioned topics automatically when enable topic level replication (#22537)
• [improve][broker] Don't log brokerClientAuthenticationParameters and bookkeeperClientAuthenticationParameters by default (#22395)
• [improve][broker] Improve Gzip compression, allow excluding specific paths or disabling it (#22370)
• [improve][broker] Optimize gzip compression for /metrics endpoint by sharing/caching compressed result (#22521)
• [improve][broker] Propagate cause exception in TopicBusyException when applicable (#22596)
• [improve][broker] Repeat the handleMetadataChanges callback when configurationMetadataStore equals localMetadataStore (#22519)
• [improve][broker] Servlet support response compression (#21667)
• [improve][broker] Support X-Forwarded-For and HA Proxy Protocol for resolving original client IP of http/https requests (#22524)
• [improve][broker] backlog quota exceed limit log replaced with debug (#22488)
• [fix][ml] Fix NPE of getValidPositionAfterSkippedEntries when recovering a terminated managed ledger (#22552)
• [fix][ml] Mark delete stuck due to switching cursor ledger fails (#22662)
• [fix][ml] No rollover inactive ledgers when metadata service invalid (#22284)
• [fix][offload] Fix OOM in tiered storage, caused by unbounded offsets cache (#22679)
• [fix][offload] Increase file upload limit from 2048MiB to 4096MiB for GCP/GCS offloading (#22554)
• [fix][storage] ReadonlyManagedLedger initialization does not fill in the properties (#22630)
• [fix][admin] Fix namespace admin api exception response (#22587)
• [fix][txn]Handle exceptions in the transaction pending ack init (#21274)
• [improve][log] Print source client addr when enabled haProxyProtocolEnabled (#22686)
• [improve][meta] Log a warning when ZK batch fails with connectionloss (#22566)
• [improve][offload] Apply autoSkipNonRecoverableData configuration to tiered storage (#22531)
• [improve][offload] Replace usage of shaded class in OffsetsCache (#22683)
• [improve][sec] Align some namespace level policy authorisation check (#21640)

Client

• [fix][client] Fix Consumer should return configured batch receive max messages (#22619)
• [fix][client] Fix ReaderBuilder doest not give illegalArgument on connection failure retry (#22639)
• [fix][client] Fix client side memory leak when call MessageImpl.create and fix imprecise client-side metrics: pendingMessagesUpDownCounter, pendingBytesUpDownCounter, latencyHistogram (#22393)

Pulsar IO and Pulsar Functions

• [fix][fn] make sure the classloader for ContextImpl is functionClassLoader in different runtimes (#22501)
• [fix][io] CompressionEnabled didn't work on elasticsearch sink (#22565)
• [fix][io] Fix es index creation (#22654) (#22701)
• [fix][io] Kafka Source connector maybe stuck (#22511)
• [improve][io]: Add validation for JDBC sink not supporting primitive schema (#22376)

...

What's Changed

Broker

• [fix][broker] Avoid execute prepareInitPoliciesCacheAsync if namespace is deleted (#22268)
• [fix][broker] Avoid expired unclosed ledgers when checking expired messages by ledger closure time (#22335)
• [fix][broker] Check cursor state before adding it to the waitingCursors (#22191)
• [fix][broker] Close dispatchers stuck due to mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22270)
• [fix][broker] Fix OpReadEntry.skipCondition NPE issue (#22367)
• [fix][broker] Fix ResourceGroup report local usage (#22340)
• [fix][broker] Fix ResourceGroups loading (#21781)
• [fix][broker] Fix issue of field 'topic' is not set when handle GetSchema request (#22377)
• [fix][broker] Fix wrong double-checked locking for readOnActiveConsumerTask in dispatcher (#22279)
• [fix][broker] fix mismatch between dispatcher.consumerList and dispatcher.consumerSet (#22283)
• [fix][ml]Expose ledger timestamp (#22338)
• [fix][offload] Fix Offload readHandle cannot close multi times. (#22162)
• [fix][txn]Fix TopicTransactionBuffer potential thread safety issue (#22149)
• [improve][admin] Fix the createMissingPartitions doesn't response correctly (#22311)
• [improve][broker] Add createTopicIfDoesNotExist option to RawReader constructor (#22264)
• [improve][broker] Add fine-grain authorization to ns/topic management endpoints (#22305)
• [improve][broker] Add missing configuration keys for caching catch-up reads (#22295)
• [improve][broker] Support create RawReader based on configuration (#22280)

Client

• [fix][client] Consumer lost message ack due to race condition in acknowledge with batch message (#22353)
• [fix][client] Do no retrying for error subscription not found when disabled allowAutoSubscriptionCreation (#22164)
• [fix][client] Fix wrong results of hasMessageAvailable and readNext after seeking by timestamp (#22363)
• [fix][client] GenericProtobufNativeSchema not implement getNativeSchema method. (#22204)
• [fix][client] Unclear error message when creating a consumer with two same topics (#22255)
• [fix][client] fix Reader.hasMessageAvailable might return true after seeking to latest (#22201)
• [fix][client]Fixed getting an incorrect maxMessageSize value when accessing multiple clusters in the same process (#22306)

Others

• [fix][misc] Make ConcurrentBitSet thread safe (#22361)
• [improve][misc] Remove the call to sun InetAddressCachePolicy (#22329)
• [fix][ws] Check the validity of config before start websocket service (#22231)

Library updates

• [fix][sec] Upgrade jose4j to 0.9.4 (#22273)
• [fix][sec] Go Functions security updates (#21844)
• [fix][sec] Upgrade Zookeeper to 3.9.2 to address CVE-2024-23944 (#22275)
• [improve][misc] Include native epoll library for Netty for arm64 (#22319)
• [improve][misc] Upgrade Netty version to 4.1.105.Final (#21923)

Tests & CI

• [fix][ci] Enable CI for branch-3.2 (#22287)
• [fix][ci] Tolerate mount option change failing in CI (#22241)
• [fix][test] Fix flaky ManagedLedgerErrorsTest.recoverAfterZnodeVersionError (#22368)
• [fix][test] Fix flaky RGUsageMTAggrWaitForAllMsgsTest (#22252)
• [fix][test] Fix flaky test BrokerServiceAutoSubscriptionCreationTest (#22190)
• [fix][test] Fix flaky test ManagedLedgerTest.testGetNumberOfEntriesInStorage (#22344)
• [fix][test] Fix thread leak in TopicPoliciesAuthZTest (#22257)
• [fix][test][branch-3.2] Fix broken ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [fix][fn] fix broken function-go test (#22260)
• [fix][test] Fix the tests with same namespace name (#22240)
• [fix][build] Fix version in pulsar-bom/pom.xml and fix src/set-project-version.sh
• [improve][test][branch-3.2] Improve ManagedLedgerTest.testGetNumberOfEntriesInStorage
• [improve][misc] Upgrade checkstyle to 10.14.2 (#22291)

For the complete list, check the full changelog.