Create ZK ancestors optimistically

[Algunenano]

Changelog category (leave one):

• Improvement

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

• Create ZK ancestors optimistically

Documentation entry for user-facing changes

---

When creating ZK ancestors, instead of going from root to last node, attempt to do it backwards as create leaves first.

For example, when creating ancestors for /clickhouse/zero_copy/zero_copy_s3/c5b2eca0-b037-49a3-89b9-3ecad815f413/all_64146_64153_1/ the old behaviour would be:

CREATE REQUEST /clickhouse
CREATE REQUEST /clickhouse/zero_copy
CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3
CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3/c5b2eca0-b037-49a3-89b9-3ecad815f413
CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3/c5b2eca0-b037-49a3-89b9-3ecad815f413/all_64146_64153_1

The new behaviour is optimistic and tries the last node first and only in case of failure it navigates the tree.

• New behaviour, best case scenario: Only one request (either it already exists or the parent exists and it's created):

CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3/c5b2eca0-b037-49a3-89b9-3ecad815f413/all_64146_64153_1

• New behaviour, worst case scenario: Whole tree.

CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3/c5b2eca0-b037-49a3-89b9-3ecad815f413/all_64146_64153_1
CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3/c5b2eca0-b037-49a3-89b9-3ecad815f413
CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3
CREATE REQUEST /clickhouse/zero_copy
CREATE REQUEST /clickhouse
CREATE REQUEST /clickhouse/zero_copy
CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3
CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3/c5b2eca0-b037-49a3-89b9-3ecad815f413
CREATE REQUEST /clickhouse/zero_copy/zero_copy_s3/c5b2eca0-b037-49a3-89b9-3ecad815f413/all_64146_64153_1

The worst case is much worse (2x requests), but the best case is much better (1 vs 5 requests). The thing is that the best case happens way more often than the worst (which never happens in fact as some parent nodes are always created earlier in the process).

This should help reducing the load in ZK caused by merges with zero copy replication, which requires an exclusive lock via ZK (tryCreateZeroCopyExclusiveLock):

• The first replicas doing the merge only has to create the last node (the part name), as the rest of the tree is already there. 5 ZK calls vs 1.
• Other replicas don't need to create anything as the last node already exists. 5 calls vs 1.

[robot-clickhouse-ci-1]

This is an automated comment for commit b8f73db with description of existing statuses. It's updated for the latest CI running
The full report is available here
The overall status of the commit is 🟡 pending

Check name	Description	Status
CI running	A meta-check that indicates the running CI. Normally, it's in success or pending state. The failed status indicates some problems with the PR	🟡 pending
Mergeable Check	Checks if all other necessary checks are successful	🟢 success
Push to Dockerhub	The check for building and pushing the CI related docker images to docker hub	🟢 success

[Algunenano]

Failures look related to the change:

helpers.client.QueryRuntimeException: Client failed! Return code: 231, stderr: Received exception from server (version 23.7.1):
E           Code: 999. DB::Exception: Received from 172.16.6.6:9000. Coordination::Exception. Coordination::Exception: Session expired, path: /d. Stack trace:
E           
E           0. ./build_docker/./contrib/llvm-project/libcxx/include/exception:134: Poco::Exception::Exception(String const&, int) @ 0x0000000043de0293 in /usr/bin/clickhouse
E           1. ./build_docker/./src/Common/Exception.cpp:94: DB::Exception::Exception(DB::Exception::MessageMasked&&, int, bool) @ 0x0000000024496d30 in /usr/bin/clickhouse
E           2. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: Coordination::Exception::Exception(String const&, Coordination::Error, int) @ 0x000000003c689933 in /usr/bin/clickhouse
E           3. ./build_docker/./contrib/llvm-project/libcxx/include/string:1499: Coordination::Exception::Exception(Coordination::Error, String const&) @ 0x000000003c68ae2e in /usr/bin/clickhouse
E           4. ./build_docker/./src/Common/ZooKeeper/ZooKeeper.cpp:0: zkutil::ZooKeeper::createAncestors(String const&) @ 0x000000003c6a00c7 in /usr/bin/clickhouse

I'm having a look.

Edit: Fixed now. It was incorrectly rejecting paths without ancestors (e.g. /t).

[Algunenano]

Failures:

• Stress test (tsan) -> Known issue Lock order inversion in OvercommitTracker, ProcessList, DiskAccessStorage, QueryStatus and Context #51150
• Upgrade check (debug) -> OOM