KVM HA Enhancements for Ceph RBD Primary Storage #5862

dhslove · 2022-01-14T05:48:30Z

Description

This PR includes the features to provide high availability using Ceph RBD-based Primary Storage.

The KVM Host HA feature uses NFS Primary Storage. However, an additional storage consideration in KVM is Ceph RBD. This PR considers the following two to provide HA using RBD.

Must use the multi-monitor server feature of Ceph RBD.
The RBD-based HA function should be provided in the same way as the NFS-based HA.

This PR includes the feature to enter multiple monitor addresses separated by commas in the RADOS Monitor item when registering RBD Primary Storage.

And it includes adding code to provide HA based on RBD Primary Storage.

To support RBD-based HA, add RBDStoragePool class to KVMHABase and use it to perform heartbeat check and activity check in the same way as NFS. Similar to how NFS writes hb and ac files to a KVMHA directory, RADOS commands are used to put hb and ac files into the RBD pool registered as primary storage.

To perform this heartbeat and activity check, kvmheartbeat_rbd.sh and kvmvmactivity_rbd.sh files were added. This is to clearly distinguish roles from the same shell file for nfs.

This PR is necessary to support agent-free HA based on Ceph RBD like NFS, even if the related PR #4978 is merged.

The execution result of the function can be checked in the attached image.

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

RBD Multiple Monitor Support - Add RBD Primary Storage

RBD Multiple Monitor Support - RBD Primary Storage List

RBD Multiple Monitor Support - KVM Host Secret List

RBD Multiple Monitor Support - VM Disk XML(Multi RBD Monitor)

RBD Based KVM Host HA - One of Hosts Off

RBD Based KVM Host HA - Host Suspect State

RBD Based KVM Host HA - Host Down State

…oing to the API

RBD primary storage 연결시 + 및 / 기호를 -와 _ 로 변경하는 로직 개선

rbd multi monitor 지원

comma로 넣을 경우 userInfo 도 null 로 출력되는 현상 수정

rbd일 때 uuid 에러로 생성로직 변경

multi monitor인 경우 로직 변경

코드 정리

Primary Storage가 RBD인 경우에 Host HA가 동작하도록 기능개선

nfsstoragepool, rbdstoragepool 분리

storagepool > pooltype 제거 및 메서드명변경(getRbdMonIpAddress)

rbd pool auth 변경가능하도록 수정

nfs, rbd Storage hb모니터링 함께 되도록 수정

sureshanaparti · 2022-01-14T11:11:00Z

@blueorangutan package

blueorangutan · 2022-01-14T11:12:04Z

@sureshanaparti a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.

blueorangutan · 2022-01-14T12:16:01Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 2208

sureshanaparti · 2022-01-14T13:00:35Z

@blueorangutan test

blueorangutan · 2022-01-14T13:01:03Z

@sureshanaparti a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

wido · 2022-01-14T15:36:10Z

This looks very interesting. I'll review this and get back. Give me a few days

blueorangutan · 2022-01-14T22:45:09Z

Trillian test result (tid-2879)
Environment: kvm-centos7 (x2), Advanced Networking with Mgmt server 7
Total time taken: 33668 seconds
Marvin logs: https://github.com/blueorangutan/acs-prs/releases/download/trillian/pr5862-t2879-kvm-centos7.zip
Smoke tests completed. 92 look OK, 0 have errors
Only failed tests results shown below:

Test	Result	Time (s)	Test File

wido · 2022-01-17T12:24:48Z

May I ask why you are not just looking up if there is a Watcher on RBD images?

dhslove · 2022-01-18T00:08:47Z

Thanks your review. @wido

The reason is

To use the same method as NFS (modified time check),
RBD Watcher continues to exist when the KVM host hangs (Watcher's inaccuracy in some cases).

...g/apache/cloudstack/storage/datastore/lifecycle/CloudStackPrimaryDataStoreLifeCycleImpl.java

.../kvm/src/main/java/com/cloud/hypervisor/kvm/resource/wrapper/LibvirtFenceCommandWrapper.java

...cloud/hypervisor/kvm/resource/wrapper/LibvirtCheckVMActivityOnStoragePoolCommandWrapper.java

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java

Rbd primary ha

코딩규칙 및 요청사항 수정

runHeartbeatToPool 메서드명 수정

HA Cloudstack 요청사항 수정

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java

wido

I am not convinced that this is the best route forward.

RBD already supports watchers and exclusive locking on images by default. If a host dies you should see within 30 sec or so if it is no longer watching a RBD image and also has not refreshed it's lock anymore.

This external script we are calling also does not support IPv6 (InetAddress) and would rely on external tooling.

Imho we should have each Host maybe just 'watch' a RBD image. Other hosts can check if there still is a watcher on that image. If not we can assume the Host has died.

Overall I think we should not merge this and rely on the things Ceph already has like exclusive locking and watchers on RBD images.

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHABase.java

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAChecker.java

scripts/vm/hypervisor/kvm/kvmheartbeat_rbd.sh

1. 각 호스트용 rbd 이미지(hb-hostip) 생성하여 watcher 감시 2. /etc/ceph 파일생성 대신 -c, -k 옵션을 사용하여 rbd 실행 3. keyring파일은 /etc/cloudstack/agent/ 경로에 생성

py파일에 파이썬 버전 설정

wido

Code looks good

DaanHoogland · 2022-12-07T07:54:27Z

@GutoVeronezi @ravening I resolved all outdated comments. cc @weizhouapache. Do you giys approve?

weizhouapache · 2022-12-08T10:22:33Z

@blueorangutan package

blueorangutan · 2022-12-08T10:23:03Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-12-08T10:37:37Z

Packaging result: ✖️ el7 ✖️ el8 ✖️ debian ✖️ suse15. SL-JID 4814

weizhouapache · 2022-12-08T10:44:10Z

@dhslove
can you solve the compilation error ?

[ERROR] COMPILATION ERROR : 
[ERROR] /home/travis/build/apache/cloudstack/plugins/storage/volume/default/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/CloudStackPrimaryDataStoreLifeCycleImpl.java:[241,16] error: cannot find symbol
  symbol:   variable multi
  location: class CloudStackPrimaryDataStoreLifeCycleImpl
[ERROR] /home/travis/build/apache/cloudstack/plugins/storage/volume/default/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/CloudStackPrimaryDataStoreLifeCycleImpl.java:[241,15] error: illegal parenthesized expression
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile (default-compile) on project cloud-plugin-storage-volume-default: Compilation failure: Compilation failure: 
[ERROR] /home/travis/build/apache/cloudstack/plugins/storage/volume/default/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/CloudStackPrimaryDataStoreLifeCycleImpl.java:[241,16] error: cannot find symbol
[ERROR]   symbol:   variable multi
[ERROR]   location: class CloudStackPrimaryDataStoreLifeCycleImpl
[ERROR] /home/travis/build/apache/cloudstack/plugins/storage/volume/default/src/main/java/org/apache/cloudstack/storage/datastore/lifecycle/CloudStackPrimaryDataStoreLifeCycleImpl.java:[241,15] error: illegal parenthesized expression
[ERROR] -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] https://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <args> -rf :cloud-plugin-storage-volume-default
The command "./tools/travis/install.sh" failed and exited with 1 during .

GutoVeronezi

@dhslove, ACS already uses java bindings for librados in com.ceph.rados.Rados; could not the heartbeat be implemented through Java instead of creating Python scripts? cc: @wido

GutoVeronezi · 2022-12-09T01:44:19Z

plugins/hypervisors/kvm/src/main/java/com/cloud/ha/KVMInvestigator.java

        for (StoragePoolVO pool : clusterPools) {
-            if (pool.getPoolType() == StoragePoolType.NetworkFilesystem) {
-                hasNfs = true;
+            if (pool.getPoolType() == StoragePoolType.NetworkFilesystem || pool.getPoolType() == StoragePoolType.RBD) {


@dhslove as these conditions are used more than once, we could create a method in StoragePoolType to do the validation. Something like this:

public boolean supportsHa() { return this.equals(NetworkFilesystem) || this.equals(RBD); }

And then:

Suggested change

if (pool.getPoolType() == StoragePoolType.NetworkFilesystem || pool.getPoolType() == StoragePoolType.RBD) {

if (pool.getPoolType().supportsHa()) {

GutoVeronezi · 2022-12-09T01:51:51Z

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHABase.java

+        String _poolUUID;
+        String _monHost;
+        String _poolMountSourcePath;
+        String _mountDestPath;
+        PoolType _type;
+        String _poolAuthUserName;
+        String _poolAuthSecret;
+        String _poolSourceHost;


These _ in the begin of the name is a legacy behavior, it should not be used anymore.

GutoVeronezi · 2022-12-09T02:06:08Z

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHABase.java

@@ -57,6 +58,28 @@ public NfsStoragePool(String poolUUID, String poolIp, String poolSourcePath, Str
        }
    }

+    public static class RbdStoragePool {
+        String _poolUUID;


IMHO, using strict camel case (non consecutive capitals/treat acronyms as normal words) would improve readability; with this approach, the variable name would be poolUuid. Also, this approach is already being applied in the class name: it is RbdStoragePool instead of RBDStoragePool.

GutoVeronezi · 2022-12-09T02:06:45Z

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHABase.java

+        String _poolAuthSecret;
+        String _poolSourceHost;
+
+        public RbdStoragePool(String poolUUID, String monHost, String poolSourcePath, String mountDestPath, PoolType type, String poolAuthUserName, String poolAuthSecret, String poolSourceHost) {


The same remark about applying camel case to acronyms.

GutoVeronezi · 2022-12-09T02:09:18Z

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAChecker.java

+        for (RbdStoragePool rbdpools : rbdStoragePools) {
+            hostAndPools = String.format("host IP [%s] in pools [%s]", hostIp, rbdStoragePools.stream().map(pool -> pool._monHost).collect(Collectors.joining(", ")));
+            s_logger.debug(String.format("Checking heart beat with KVMHAChecker for %s", hostAndPools));
+
+            ProcessBuilder processBuilder = new ProcessBuilder();
+            processBuilder.command().add("python3");
+            processBuilder.command().add(s_heartBeatPathRbd);
+            processBuilder.command().add("-i");
+            processBuilder.command().add(rbdpools._poolSourceHost);
+            processBuilder.command().add("-p");
+            processBuilder.command().add(rbdpools._poolMountSourcePath);
+            processBuilder.command().add("-n");
+            processBuilder.command().add(rbdpools._poolAuthUserName);
+            processBuilder.command().add("-s");
+            processBuilder.command().add(rbdpools._poolAuthSecret);
+            processBuilder.command().add("-v");
+            processBuilder.command().add(hostIp);
+            processBuilder.command().add("-r");
+            processBuilder.command().add("r");
+            Process process = null;
+            String parsedLine = "";
+            try {
+                process = processBuilder.start();
+                BufferedReader bfr = new BufferedReader(new InputStreamReader(process.getInputStream()));
+                parsedLine = bfr.readLine();
+            } catch (IOException e) {
+                e.printStackTrace();
+            }
+
+            s_logger.debug(String.format("Checking heart beat with KVMHAChecker [{command=\"%s\", log: \"%s\", pool: \"%s\"}].", processBuilder.command().toString(), parsedLine,
+            rbdpools._monHost));
+
+            if (process != null && parsedLine.contains("DEAD")) {
+                s_logger.warn(String.format("Checking heart beat with KVMHAChecker command [%s] returned. [%s]. It may cause a shutdown of host IP [%s].", processBuilder.command().toString(),
+                        parsedLine, hostIp));
+            } else {
+                validResult = true;
+            }
+        }
+


This block could be extracted to a method, to better organize the code.

GutoVeronezi · 2022-12-09T02:38:58Z

.../hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAVMActivityChecker.java

+            ProcessBuilder processBuilder = new ProcessBuilder();
+            processBuilder.command().add("python3");
+            processBuilder.command().add(vmActivityCheckPath);
+            processBuilder.command().add("-i");
+            processBuilder.command().add(rbdStoragePool._poolSourceHost);
+            processBuilder.command().add("-p");
+            processBuilder.command().add(rbdStoragePool._poolMountSourcePath);
+            processBuilder.command().add("-n");
+            processBuilder.command().add(rbdStoragePool._poolAuthUserName);
+            processBuilder.command().add("-s");
+            processBuilder.command().add(rbdStoragePool._poolAuthSecret);
+            processBuilder.command().add("-v");
+            processBuilder.command().add(hostIp);
+            processBuilder.command().add("-u");
+            processBuilder.command().add(volumeUuidList);
+            command = processBuilder.command().toString();
+            Process process = null;
+
+            try {
+                process = processBuilder.start();
+                BufferedReader bfr = new BufferedReader(new InputStreamReader(process.getInputStream()));
+                parsedLine = bfr.readLine();
+            } catch (IOException e) {
+                e.printStackTrace();
+            }


Same here about using com.cloud.utils.script.Script.

GutoVeronezi · 2022-12-09T02:40:11Z

...cloud/hypervisor/kvm/resource/wrapper/LibvirtCheckVMActivityOnStoragePoolCommandWrapper.java

@@ -43,9 +44,16 @@ public Answer execute(final CheckVMActivityOnStoragePoolCommand command, final L
        final ExecutorService executors = Executors.newSingleThreadExecutor();
        final KVMHAMonitor monitor = libvirtComputingResource.getMonitor();
        final StorageFilerTO pool = command.getPool();
-        if (Storage.StoragePoolType.NetworkFilesystem == pool.getType()){
+        if (Storage.StoragePoolType.NetworkFilesystem == pool.getType() || Storage.StoragePoolType.RBD == pool.getType()){


If created, the method supportsHa could be used here too.

GutoVeronezi · 2022-12-09T02:40:38Z

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

-                final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.RESET, null);
-                return resp.getSuccess();
+                final OutOfBandManagement oobm = outOfBandManagementDao.findByHost(r.getId());
+                if(oobm.getPowerState() == PowerState.Off){


Suggested change

if(oobm.getPowerState() == PowerState.Off){

if (oobm.getPowerState() == PowerState.Off) {

GutoVeronezi · 2022-12-09T02:40:49Z

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

+                if(oobm.getPowerState() == PowerState.Off){
+                    LOG.warn("OOBM recover operation failed for the host " + r.getName() + " already OFF");
+                    return false;
+                }else{


Suggested change

}else{

} else {

GutoVeronezi · 2022-12-09T02:41:08Z

plugins/hypervisors/kvm/src/main/java/org/apache/cloudstack/kvm/ha/KVMHAProvider.java

-                final OutOfBandManagementResponse resp = outOfBandManagementService.executePowerOperation(r, PowerOperation.OFF, null);
-                return resp.getSuccess();
+                final OutOfBandManagement oobm = outOfBandManagementDao.findByHost(r.getId());
+                if (oobm.getPowerState() == PowerState.Unknown){


Suggested change

if (oobm.getPowerState() == PowerState.Unknown){

if (oobm.getPowerState() == PowerState.Unknown) {

rbd primary ha 추가사항 반영

github-actions · 2023-02-27T05:25:41Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

shwstppr · 2023-10-03T09:19:10Z

@dhslove can you please address the merge conflicts? Should we consider this for 4.19.0.0?

weizhouapache · 2024-09-25T09:58:47Z

moved to 4.21.0.0 milestone

ycyun and others added 13 commits October 20, 2021 17:05

Replace multiple + and / symbols by - and _ to have URL-safe base64 g…

3bfbbc8

…oing to the API

Merge pull request #91 from ycyun/main

ecb312d

RBD primary storage 연결시 + 및 / 기호를 -와 _ 로 변경하는 로직 개선

rbd multi monitor 지원

c4f9c6e

rbd multi monitor 지원

comma로 넣을 경우 userInfo 도 null 로 출력되는 현상 수정

2026348

comma로 넣을 경우 userInfo 도 null 로 출력되는 현상 수정

rbd일 때 uuid 에러로 생성로직 변경

a2a12cd

rbd일 때 uuid 에러로 생성로직 변경

multi monitor인 경우 로직 변경

8509679

multi monitor인 경우 로직 변경

코드 정리

3f3ccdb

코드 정리

Host HA : RBD Primary Storage에 대한 Host HA 기능 제공

45c761c

Primary Storage가 RBD인 경우에 Host HA가 동작하도록 기능개선

Host HA : RBD Primary Storage에 대한 Host HA 기능 제공

eb953b5

nfsstoragepool, rbdstoragepool 분리

Host HA : RBD Primary Storage에 대한 Host HA 기능 제공

c1ea010

storagepool > pooltype 제거 및 메서드명변경(getRbdMonIpAddress)

Host HA : RBD Primary Storage에 대한 Host HA 기능 제공

481afe7

rbd pool auth 변경가능하도록 수정

Host HA : RBD Primary Storage에 대한 Host HA 기능 제공

4dcd141

nfs, rbd Storage hb모니터링 함께 되도록 수정

Merge branch 'main' into rbd-primary-ha

3b14b8d

GutoVeronezi reviewed Jan 26, 2022

View reviewed changes

sjyu1 and others added 4 commits January 28, 2022 14:23

Merge pull request #5 from ablecloud-team/rbd-primary-ha

c4f2ba7

Rbd primary ha

HA Cloudstack 요청사항 수정

9c80a17

코딩규칙 및 요청사항 수정

HA Cloudstack 요청사항 수정

8a71e48

runHeartbeatToPool 메서드명 수정

Merge pull request #147 from sjyu1/rbd-primary-ha

81755fc

HA Cloudstack 요청사항 수정

DaanHoogland reviewed Jan 31, 2022

View reviewed changes

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java Outdated Show resolved Hide resolved

plugins/hypervisors/kvm/src/main/java/com/cloud/hypervisor/kvm/resource/KVMHAMonitor.java Outdated Show resolved Hide resolved

wido reviewed Jan 31, 2022

View reviewed changes

rbd watcher 방식으로 변경

29a544c

1. 각 호스트용 rbd 이미지(hb-hostip) 생성하여 watcher 감시 2. /etc/ceph 파일생성 대신 -c, -k 옵션을 사용하여 rbd 실행 3. keyring파일은 /etc/cloudstack/agent/ 경로에 생성

Merge pull request #255 from sjyu1/rbd-primary-ha2

b5c57af

py파일에 파이썬 버전 설정

wido approved these changes Dec 6, 2022

View reviewed changes

GutoVeronezi reviewed Dec 9, 2022

View reviewed changes

sjyu1 and others added 5 commits December 16, 2022 13:18

multi monitor 소스삭제 및 관련파일 수정

f094dff

poolUUID를 poolUuid로 카멜케이스 적용

bbabc99

if문 space 적용

f64ad6b

supportHa () 메소드 추가

5323658

Merge pull request #272 from sjyu1/rbd-primary-ha01

21cb3c9

rbd primary ha 추가사항 반영

boring-cyborg bot added component:build type:security labels Feb 27, 2023

github-actions bot added the status:has-conflicts label Feb 27, 2023

DaanHoogland added this to the 4.19.0.0 milestone Jun 22, 2023

weizhouapache mentioned this pull request Oct 5, 2023

KVM Host HA enhancement for StorPool storage #8045

Merged

8 tasks

shwstppr closed this Oct 9, 2023

shwstppr reopened this Oct 9, 2023

DaanHoogland modified the milestones: 4.19.0.0, unplanned Nov 15, 2023

weizhouapache mentioned this pull request Mar 20, 2024

KVM Host-HA using CEPH RBD #8805

Closed

philtreF mentioned this pull request Sep 25, 2024

[Status] KVM Host-HA using CEPH RBD #9728

Open

weizhouapache modified the milestones: unplanned, 4.21.0.0 Sep 25, 2024

DaanHoogland linked an issue Sep 25, 2024 that may be closed by this pull request

[Status] KVM Host-HA using CEPH RBD #9728

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

KVM HA Enhancements for Ceph RBD Primary Storage #5862

KVM HA Enhancements for Ceph RBD Primary Storage #5862

dhslove commented Jan 14, 2022

sureshanaparti commented Jan 14, 2022

blueorangutan commented Jan 14, 2022

blueorangutan commented Jan 14, 2022

sureshanaparti commented Jan 14, 2022

blueorangutan commented Jan 14, 2022

wido commented Jan 14, 2022

blueorangutan commented Jan 14, 2022

wido commented Jan 17, 2022

dhslove commented Jan 18, 2022 •

edited

Loading

wido left a comment

wido left a comment

DaanHoogland commented Dec 7, 2022

weizhouapache commented Dec 8, 2022

blueorangutan commented Dec 8, 2022

blueorangutan commented Dec 8, 2022

weizhouapache commented Dec 8, 2022

GutoVeronezi left a comment •

edited

Loading

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

GutoVeronezi Dec 9, 2022

github-actions bot commented Feb 27, 2023

shwstppr commented Oct 3, 2023

weizhouapache commented Sep 25, 2024

	if (pool.getPoolType() == StoragePoolType.NetworkFilesystem \|\| pool.getPoolType() == StoragePoolType.RBD) {
	if (pool.getPoolType().supportsHa()) {

	if(oobm.getPowerState() == PowerState.Off){
	if (oobm.getPowerState() == PowerState.Off) {

	if (oobm.getPowerState() == PowerState.Unknown){
	if (oobm.getPowerState() == PowerState.Unknown) {

KVM HA Enhancements for Ceph RBD Primary Storage #5862

Are you sure you want to change the base?

KVM HA Enhancements for Ceph RBD Primary Storage #5862

Conversation

dhslove commented Jan 14, 2022

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

sureshanaparti commented Jan 14, 2022

blueorangutan commented Jan 14, 2022

blueorangutan commented Jan 14, 2022

sureshanaparti commented Jan 14, 2022

blueorangutan commented Jan 14, 2022

wido commented Jan 14, 2022

blueorangutan commented Jan 14, 2022

wido commented Jan 17, 2022

dhslove commented Jan 18, 2022 • edited Loading

wido left a comment

Choose a reason for hiding this comment

wido left a comment

Choose a reason for hiding this comment

DaanHoogland commented Dec 7, 2022

weizhouapache commented Dec 8, 2022

blueorangutan commented Dec 8, 2022

blueorangutan commented Dec 8, 2022

weizhouapache commented Dec 8, 2022

GutoVeronezi left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot commented Feb 27, 2023

shwstppr commented Oct 3, 2023

weizhouapache commented Sep 25, 2024

dhslove commented Jan 18, 2022 •

edited

Loading

GutoVeronezi left a comment •

edited

Loading