Add volume group handling #6012

DK101010 · 2022-02-21T08:35:25Z

Description

Motivation:
Some applications needs an specific volume controller configuration. For example to get a optimal performance or it is necessary to get an certificate.
Therefore admins needs a opportunity to add an volume to certain controller.

Implementation:
Decided to add a VolumeGroup class as an abstract object to manage this feature.
A volume group is a triple of vm id, volume id and group number.

group number == bus number
groups are optional, user can work like before
groups can only created via api
during the attach the group will be create
if vm have got mixed volumes with and without groups,
attach order is: root vol, vol with groups(order by disk seq), vols without groups(order by disk seq).
for import unmanaged VMs user have got three option:
1. can do like before (no groups) => import with current conf, after restart volumes
will be attached on the first controller until max and then second controller and so on.
(applied after restart)
2. add vol groups to disk offering list => add groups like user want it (applied after restart)
3. set useControllerConfiguration = true => read current controller conf of unmanaged vm
and add automatic volumegroups for each volume

SQL for Update:

DROP TABLE IF EXISTS `cloud`.`volume_group`;
CREATE TABLE IF NOT EXISTS `cloud`.`volume_group`(
    `id` bigint unsigned PRIMARY KEY AUTO_INCREMENT COMMENT 'Primary Key',
    `vm_id` bigint unsigned NOT NULL COMMENT 'foreign key to vm table',
    `volume_id` bigint unsigned NOT NULL COMMENT 'foreign key to volume table',
    `group_number` int unsigned NOT NULL COMMENT 'group number of volume',
    FOREIGN KEY(vm_id) REFERENCES vm_instance (id),
    FOREIGN KEY(volume_id) REFERENCES volumes (id)
);

Types of changes

Breaking change (fix or feature that would cause existing functionality to change)
New feature (non-breaking change which adds functionality)
Bug fix (non-breaking change which fixes an issue)
Enhancement (improves an existing feature and functionality)
Cleanup (Code refactoring and cleanup, that may add test cases)

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Major
Minor

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

DaanHoogland

like the idea,
is this (meant to be) vmware only?

api/src/main/java/org/apache/cloudstack/api/command/user/volume/AttachVolumeCmd.java

DK101010 · 2022-02-21T09:48:33Z

like the idea, is this (meant to be) vmware only?

In the future also for KVM but for now only for vmware.

DK101010 · 2022-03-22T08:21:12Z

Hi @DaanHoogland, @nvazquez,
I have a question, what I have to do to add follow code

def importUnmanagedInstance(self, command, method="GET"):
         response = importUnmanagedInstanceResponse()
         response = self.connection.marvinRequest(command, response_type=response, method=method)
         return response

to

cloudstack/tools/marvin/marvin/cloudstack/cloudstackAPIClient.py

I added in my env, but it seems to be automated generated or ?

DaanHoogland · 2022-03-22T12:21:51Z

Hi @DaanHoogland, @nvazquez, I have a question, what I have to do to add follow code
cloudstack/tools/marvin/marvin/cloudstack/cloudstackAPIClient.py
I added in my env, but it seems to be automated generated or ?

yes, it should be auto generated.

nvazquez · 2022-03-22T12:40:36Z

Hi @DK101010 - just extending @DaanHoogland's answer. It should be generated by building marvin: mvn -P developer -pl :cloud-marvin and upgrade the packages pip install --upgrade tools/marvin/dist/Marvin-*.tar.gz

DK101010 · 2022-03-22T13:27:52Z

Ahh, ok a case of self confusion :D. Thanks you two

github-actions · 2022-04-07T05:56:51Z

Hi @${author}, your pull request has merge conflicts. Can you fix the conflicts and sync your branch with the base branch?

DK101010 · 2022-04-12T10:17:57Z

Soo, I guess I'm finished and ready for review :)

DaanHoogland · 2022-04-13T06:51:32Z

Soo, I guess I'm finished and ready for review :)

you mean you are taking it out of draft @DK101010 ?

DK101010 · 2022-04-13T06:59:55Z

Soo, I guess I'm finished and ready for review :)

you mean you are taking it out of draft @DK101010 ?

Japp, correct.

DaanHoogland

some code convention comments, but cltgm.
It has integration tests so I think only regression tests are needed.

api/src/main/java/org/apache/cloudstack/api/command/admin/vm/ImportUnmanagedInstanceCmd.java

server/src/main/java/com/cloud/storage/VolumeApiServiceImpl.java

vmware-base/src/main/java/com/cloud/hypervisor/vmware/mo/VirtualMachineMO.java

DaanHoogland · 2022-04-13T07:14:43Z

Soo, I guess I'm finished and ready for review :)

you mean you are taking it out of draft @DK101010 ?

Japp, correct.

😁 than do so ;)

DK101010 · 2022-04-13T08:39:40Z

Soo, I guess I'm finished and ready for review :)

you mean you are taking it out of draft @DK101010 ?

Japp, correct.

😁 than do so ;)

Oh, how I can do it? :D I can only editing the description and the headline.

DaanHoogland · 2022-04-13T08:47:39Z

Soo, I guess I'm finished and ready for review :)

you mean you are taking it out of draft @DK101010 ?

Japp, correct.

grin than do so ;)

Oh, how I can do it? :D I can only editing the description and the headline.

You don't see the Ready for review button below here?

DaanHoogland · 2022-04-13T08:50:35Z

also @DK101010 , if you go to files you should be able to commit multiple suggestions in one commit.

DK101010 · 2022-04-13T11:43:49Z

also @DK101010 , if you go to files you should be able to commit multiple suggestions in one commit.

Thanks for the hint, I will try it next time. Or I could squash the commits if there are too many.

DaanHoogland · 2022-04-14T10:21:40Z

@blueorangutan package

DaanHoogland · 2022-10-19T12:05:11Z

I have no more ideas except to wait until 4.18 comes out.

I have built up my environment again with 4.16. That worked without any problems.

Did you report all the issues you are facing?

DK101010 · 2022-10-19T12:12:25Z

I have no more ideas except to wait until 4.18 comes out.
I have built up my environment again with 4.16. That worked without any problems.

Did you report all the issues you are facing?

ohm nope, make it sense to report it when it occur in the main? And when yes where I should report it?

DaanHoogland · 2022-10-19T12:15:50Z

If there are obvious errors in any branch report them in the issues. We can then discuss and fix or help if it is environmental.

DK101010 · 2022-10-19T12:46:55Z

If there are obvious errors in any branch report them in the issues. We can then discuss and fix or help if it is environmental.

#6837

DaanHoogland · 2022-10-19T12:56:42Z

so it was only on main, not on 4.17?

DK101010 · 2022-10-19T12:59:06Z

so it was only on main, not on 4.17?

I have only main tested and my last stable base dev branch was 4.16

DaanHoogland · 2022-12-13T08:05:46Z

@GutoVeronezi @weizhouapache can you review?

weizhouapache · 2022-12-16T08:52:48Z

@blueorangutan package

blueorangutan · 2022-12-16T08:53:04Z

@weizhouapache a Jenkins job has been kicked to build packages. It will be bundled with KVM, XenServer and VMware SystemVM templates. I'll keep you posted as I make progress.

blueorangutan · 2022-12-16T11:12:07Z

Packaging result: ✔️ el7 ✔️ el8 ✔️ debian ✔️ suse15. SL-JID 4985

DaanHoogland · 2022-12-16T13:49:38Z

@blueorangutan test

blueorangutan · 2022-12-16T13:50:03Z

@DaanHoogland a Trillian-Jenkins test job (centos7 mgmt + kvm-centos7) has been kicked to run smoke tests

blueorangutan · 2022-12-16T14:30:25Z

Trillian Build Failed (tid-5531)

blueorangutan · 2022-12-19T10:01:14Z

Trillian Build Failed (tid-5555)

DaanHoogland · 2022-12-19T11:00:54Z

@DK101010 the smoke test runs fail to start the system VMs due to

2022-12-19 09:37:50,085 WARN  [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-2:ctx-48cb44d7 job-4/job-30 ctx-f8c501d2) (logid:d647b5b8) Unable to orchestrate start VM instance {"id":2,"instanceName":"s-2-VM","type":"SecondaryStorageVm","uuid":"47535f32-97f4-4ad0-9393-64154953ac78"} due to [DB Exception on: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT volume_group.id, volume_group.vm_id, volume_group.volume_id, volume_group.group_number FROM volume_group WHERE volume_group.vm_id = 2  AND volume_group.volume_id = 2  ORDER BY RAND() LIMIT 1].
com.cloud.utils.exception.CloudRuntimeException: DB Exception on: com.mysql.cj.jdbc.ClientPreparedStatement: SELECT volume_group.id, volume_group.vm_id, volume_group.volume_id, volume_group.group_number FROM volume_group WHERE volume_group.vm_id = 2  AND volume_group.volume_id = 2  ORDER BY RAND() LIMIT 1
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:424)
        at com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoBase.java:360)
        at com.cloud.utils.db.GenericDaoBase.findOneIncludingRemovedBy(GenericDaoBase.java:897)
        at com.cloud.utils.db.GenericDaoBase.findOneBy(GenericDaoBase.java:906)
        at com.cloud.storage.dao.VolumeGroupDaoImpl.findByVmAndVolume(VolumeGroupDaoImpl.java:67)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:344)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:198)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
        at com.cloud.utils.db.TransactionContextInterceptor.invoke(TransactionContextInterceptor.java:34)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:175)
        at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:97)
        at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:186)
        at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:215)
        at com.sun.proxy.$Proxy230.findByVmAndVolume(Unknown Source)
        at com.cloud.hypervisor.HypervisorGuruBase.lambda$addGroupsToDisks$0(HypervisorGuruBase.java:313)
        at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
        at com.cloud.hypervisor.HypervisorGuruBase.addGroupsToDisks(HypervisorGuruBase.java:312)
        at com.cloud.hypervisor.HypervisorGuruBase.toVirtualMachineTO(HypervisorGuruBase.java:256)
        at com.cloud.hypervisor.KVMGuru.implement(KVMGuru.java:116)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:1205)
        at com.cloud.vm.VirtualMachineManagerImpl.orchestrateStart(VirtualMachineManagerImpl.java:5333)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)
        at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
        at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5457)
        at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:620)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:48)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:55)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:102)
        at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:52)
        at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:45)
        at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:568)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
        at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
        at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: java.sql.SQLSyntaxErrorException: Table 'cloud.volume_group' doesn't exist
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:120)
        at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:97)
        at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
        ... 48 more

can you have a look?

DaanHoogland · 2022-12-30T12:14:07Z

@DK101010 are you actively pursuing this? I will move it to the next milestone if not.

DK101010 · 2023-01-11T10:25:37Z

@DK101010 are you actively pursuing this? I will move it to the next milestone if not.

@DaanHoogland please move it, I'm currently working on other projects.

github-actions · 2023-01-17T14:46:52Z

This pull request has merge conflicts. Dear author, please fix the conflicts and sync your branch with the base branch.

shwstppr · 2023-10-03T09:19:22Z

@DK101010 can you please address the merge conflicts? Should we consider this for 4.19.0.0?

DK101010 · 2023-10-06T06:04:52Z

@shwstppr I'm sorry, but I don't have time for that at the moment. I would rather see the 4.20

shwstppr · 2023-10-06T06:25:57Z

@DK101010 thanks for the update. Moving it to 4.20.0.0 milestone

DaanHoogland marked this pull request as draft February 21, 2022 09:15

DaanHoogland reviewed Feb 21, 2022

View reviewed changes

api/src/main/java/org/apache/cloudstack/api/command/user/volume/AttachVolumeCmd.java Outdated Show resolved Hide resolved

nvazquez added this to the 4.18.0.0 milestone Mar 21, 2022

nvazquez added the status:proof-of-concept label Mar 21, 2022

github-actions bot added the status:has-conflicts label Apr 7, 2022

boring-cyborg bot added component:api component:integration-test component:marvin component:orchestration component:storage component:vmware Python Warning... Python code Ahead! labels Apr 12, 2022

DaanHoogland approved these changes Apr 13, 2022

View reviewed changes

DK101010 marked this pull request as ready for review April 13, 2022 08:48

DK101010 force-pushed the feat/add_volume_groups branch from a7a5c72 to 9c42919 Compare April 14, 2022 08:30

apache deleted a comment from blueorangutan Dec 16, 2022

DaanHoogland modified the milestones: 4.18.0.0, 4.19.0.0 Jan 11, 2023

github-actions bot added the status:has-conflicts label Jan 17, 2023

shwstppr modified the milestones: 4.19.0.0, 4.20.0.0 Oct 6, 2023

JoaoJandre modified the milestones: 4.20.0.0, 4.21.0.0 Sep 10, 2024

Add volume group handling #6012

Are you sure you want to change the base?

Add volume group handling #6012

Conversation

DK101010 commented Feb 21, 2022 • edited Loading

Description

Types of changes

Feature/Enhancement Scale or Bug Severity

Feature/Enhancement Scale

Bug Severity

Screenshots (if appropriate):

How Has This Been Tested?

DaanHoogland left a comment

Choose a reason for hiding this comment

DK101010 commented Feb 21, 2022

DK101010 commented Mar 22, 2022

DaanHoogland commented Mar 22, 2022

nvazquez commented Mar 22, 2022

DK101010 commented Mar 22, 2022

github-actions bot commented Apr 7, 2022

DK101010 commented Apr 12, 2022

DaanHoogland commented Apr 13, 2022

DK101010 commented Apr 13, 2022

DaanHoogland left a comment

Choose a reason for hiding this comment

DaanHoogland commented Apr 13, 2022

DK101010 commented Apr 13, 2022

DaanHoogland commented Apr 13, 2022

DaanHoogland commented Apr 13, 2022

DK101010 commented Apr 13, 2022

DaanHoogland commented Apr 14, 2022

DaanHoogland commented Oct 19, 2022

DK101010 commented Oct 19, 2022

DaanHoogland commented Oct 19, 2022

DK101010 commented Oct 19, 2022

DaanHoogland commented Oct 19, 2022

DK101010 commented Oct 19, 2022 • edited Loading

DaanHoogland commented Dec 13, 2022

weizhouapache commented Dec 16, 2022

blueorangutan commented Dec 16, 2022

blueorangutan commented Dec 16, 2022

DaanHoogland commented Dec 16, 2022

blueorangutan commented Dec 16, 2022

blueorangutan commented Dec 16, 2022

blueorangutan commented Dec 19, 2022

DaanHoogland commented Dec 19, 2022

DaanHoogland commented Dec 30, 2022

DK101010 commented Jan 11, 2023

github-actions bot commented Jan 17, 2023

shwstppr commented Oct 3, 2023

DK101010 commented Oct 6, 2023

shwstppr commented Oct 6, 2023

DK101010 commented Feb 21, 2022 •

edited

Loading

DK101010 commented Oct 19, 2022 •

edited

Loading