Define a release schedule for the project

[JoaoJandre]

Hi all,

I had created a thread for this on the dev mailing list, but following @DaanHoogland 's advice I've decided to create a new one here. That being said, I have a proposal for the versioning scheme. Instead of dropping the "X"
on our X.Y.Z.N, we can set a fixed schedule (that can be further discussed) for the version changes:

• Every two years, we release a major version (X), which can contain changes that break backward compatibility, such as (but not limited to): removing unused/useless APIs, renaming APIs, and changing the default behavior of features. These changes must be discussed with the devs and be properly communicated to the community (via API deprecation, for example) at least one minor version before the major release;
• Every semester, we release a minor version (Y) of the current major (X) version, these can contain new features/APIs, as long as the backward compatibility is maintained; also, feature/API deprecation should happen on these versions;
• Every 2/3 months, we release patch versions (Z), that fix any bugs that were released with the major and minor versions, these versions should not contain any new features;
• In extreme cases (such as with security issues) we work on and release
  security versions (N);

The proposed schedule is only a sketch that can be worked on. However I believe that the project can benefit from:

1. A fixed release schedule;
2. A mechanism to introduce disruptive changes, so that the project can evolve and not be chained to the same features/API/limitation/technical debts forever.

Furthermore, having a schedule allows us to have a project roadmap, that outlines the future deprecations, changes and big features.

[GutoVeronezi]

Hello guys,

Thanks @JoaoJandre for raising the discussion. Many might understand the benefits and the necessity of having a well-defined versioning schema; however, I feel that it is important to highlight the "why" before discussing the "how".

We (the community) have a culture of generating code without breaking compatibility. At first look, this may seem interesting as there is an intention to guarantee the continuity of the ecosystem. However, along with that, there is also a tie with old and deprecated procedures, technologies, libraries, and models, which, in the long term, will also deprecate CloudStack and make it more costly to innovate and evolve.

We have to understand that we have several development barriers. We are already working with a highly complex context; to work with CloudStack, developers need to understand storage, networking, virtualization, Internet protocols, and much more. On top of that we do not use mainstream/standardized frameworks for the RESTful APIs and JPA; we use Spring in a non-standard fashion; we do not have a solid standard between our APIs, and so on. We are constantly increasing the complexity to extend/evolve and maintain ACS.

That creates a bubble of development where it is hard to join, which is not healthy for a community. All those development fronts mentioned require a lot of effort and investment to get addressed and will certainly generate incompatibility at some point. However, we do not have a well-defined protocol to address incompatibilities nor to release new versions; we have been on version 4 since 2012 and our minor release cadence has been sporadic since then.

Knowing that a lot of effort and investment is required to make CloudStack compliant with new and modern standards and frameworks, and our versioning schema is not foreseeable nor addresses introducing incompatibilities between versions, would a contributor or company feel safe to sponsor and put efforts into such changes?

Unfortunately, the answer is no. With that being said, by structuring a versioning schema, we can provide previsibility for the project and the community. We would know when to introduce breaking changes, and the community would be expecting them. Furthermore, we can look at how other projects, like OpenStack and GitLab, introduce breaking changes in two steps (deprecation on one major release and removal on another) to reduce their impact on the community. Moreover, we already had a discussion in the past about the same topic; thus, I invite you to check that discussion too: https://lists.apache.org/thread/lwlxs8xgz4glocctf7dv89k5nqqsxmlb

[DaanHoogland]

@GutoVeronezi points to the more abstract reasons we want to allow breaking changes.
We have long had a list of more mondain reasons to introduce breaking changes:

• API truly restful
• ORM/DAOs upgradable per feature PR instead of per version
• agent API more consistent with state of the art json frameworks.
• ...
  and the discussion will coming back. (thanks for pointing to one of the threads that highlighted this @GutoVeronezi )

On one hand, I think some of these can be introduced without breaking as well. APIs can be running alongside of each other and be slowly phased out.
On the other hand, I'd love to have the luxury to break stuff and rebuild it, but works needs to get done. The custom DAO framework we maintain will be a pain to get rid of.

As to @JoaoJandre 's proposal:

• A fixed release schedule; I think this is not feasible in a volunteer based project. Company priorities will get in the way. We can however set a guideline for LTS branches that for instance 10+ PRs/fixes warrant a release (we now do 200+ usually, so just being arbitrary)
• A mechanism to introduce disruptive changes; Yes, we should allow for that and help users on a consultancy basis if they want to migrate.

(still think we should drop the 4 yesterday)

[JoaoJandre]

@DaanHoogland, about the points you raised:

• We have examples of fixed release schedules on volunteer based projects. I think that a great example is OpenStack (see https://releases.openstack.org/), which has a major release each 6 months. Compared to that, I think that the schedule I proposed is quite tame, with majors every two years only. The biggest thing we would have to get used to is features not getting into their intended release every time: if the feature is not ready and it's time for a new release… too bad, we leave it for the next one.
• Do you have a proposal for a mechanism to introduce disruptive changes? It seems like we all agree that it should exist, but no one wants to suggest one. I think that the natural solution would be to start following the semantic versioning that our project supposedly uses.

[DaanHoogland]

I like the last sentence of your first point. We still have a 2 releases per year policy though we never make that.

As for the second point I think we should reduce out version to a 3 number scheme (still drop the 4) and allow any changes on the basis of this. However, we should still allow deprecation periods for dropping backwards compatible components or structures. And always provide well tested migration paths.

[rohityadavcloud]

I wouldn't consider Openstack volunteer-based project. Two major releases a year, two minor releases a year is something we already aim for but struggle to "ensure" it happens.

Practically as and when you or anybody has an initiative discuss that on dev@ ML, build support and consensus and make it happen.

[rohityadavcloud]

Github Discussions isn't the right place to discuss project governance related matters, but may be used for discussions and to build initial consensus; it was enabled only as a platform to support user-forum interactions. If the intention is/was to only build consensus, then fine - but many PMCs and other stakeholders aren't on Github (Discussions or follow activities on it actively). Releases, project policy for changes, modernising codebase, I believe there they must be discussed on the dev@ ML.

Personally, I think we've well-defined project bylaws and release process that works for me (and perhaps many of us) and I'm not very interested 'currently' to change them, but on convincing arguments I (and others) may participate and help support proposed changes. We generally target two major and two minor releases every year for the CloudStack project and sub-projects releases are on need-basis. However, even if a hard policy of doing X releases a year was there - who would enforce them, who would drive them? And what happens if we as community miss that, I think there is no way to "ensure" or "enforce" that.

I appreciate the intentions but it's not practical to pull off or to "ensure" it happens even if there was any policy, protocol, what have you, in place whether it's around releases or disruptive changes. Nobody is stopping any PMC or committers to do a release or lead an initiative; but releases cost time, energy and bandwidth, so many of cannot commit full-time on this to "ensure" they happen but only make our best attempt. I wouldn't consider if Joao or Daniel or others want to do more releases, go make it happen and update https://cwiki.apache.org/confluence/display/CLOUDSTACK/Releases

There are whole variety of good remarks and thoughts in some other comments, but they don't converge on specific issues related to release schedule. However, I sense some new features or improvements can enable better DB upgrades via (new) framework/tooling that could assist better release management and upgrades. The following is my views on some of the things mentioned;

CloudStack indeed can benefit from codebase modernisation, and we're already doing that. But it cannot be replaced overnight, such changes can only be iterative and discussed on case-by-case basis as a separate initiative/proposal/idea. Some changes may be beneficial but we would need to be careful with changes that disrupt users upgrades, integrations and expectations. Community is generally good at maintenance but not with new features or disrupting the codebase in its entirety, which would require willingness of sponsors and employers to fund their people's time and bandwidth on such projects.

While we're still on 4.x, CloudStack of 2024 isn't same as that of 2012, it's a massively different, polished and improved project & product with more room to go. I don't think it's a bad thing. With what other opensource communities like Python have done with the whole v2 vs v3 transition (and they continue to do that within 3.10 and 3.10+), it's much better we don't disrupt the product too much for our users. It has both the legacy and pedigree of a product, with its features, benefits and issues.

The project is in its plateau of productivity so this isn't surprising we aren't changing much, many of us largely users may not want major disruptions, for users CloudStack must just continue work and continue to be boring. What is practical for the community will most likely happen anyway, we must continue to exercise patience, develop soft skills, learn to build consensus and support and work with others to make such ideas happen. Coding is a small part in what it takes to drive such changes and make them sustainable and acceptable for the wider community.

I think any large project such as ours, in however, whatever standards it must use is difficult to onboard new developers, we aren't unique in that way. For supporting CloudStack development learning/training, we have a self-learning courseware for aspiring developers. That said, I would encourage anything that helps improve CloudStack adoptability by developers, but it's the users I tend to include more towards.

[GutoVeronezi]

Github Discussions is integrated with the users' mailing list; everything that is posted here is also sent by email and vice-versa. This discussion is way wider than the development scope (as @rohityadavcloud stated), it affects users as well; thus, GitHub Discussions seems the right place to have it.

I see we agree on several topics, but not so much on others. What draws my attention in the whole discussion is that people say that we have a well-defined process: two major and two minor releases a year; however, with a remark that we never make that. In fact, looking at our history, we never accomplished that (regarding the major releases). Following there is the tag creation date of major releases since 4.10 (the tag creation divergences in an average of a week from the release itself):

tags creation date

for i in {10..19}; do echo $i && git show 4.$i.0.0 | head -n 3 | grep Date; done
10
Date:   Thu Jul 6 15:52:45 2017 +0530
11
Date:   Fri Jan 26 13:13:44 2018 +0100
12
Date:   Thu Mar 14 10:11:46 2019 -0300
13
Date:   Fri Sep 20 14:03:47 2019 +0530
14
Date:   Mon May 11 15:03:30 2020 +0100
15
Date:   Mon Jan 11 13:48:17 2021 +0530
16
Date:   Thu Nov 4 14:14:57 2021 -0300
17
Date:   Tue May 31 14:33:47 2022 -0300
18
Date:   Sat Mar 11 09:36:25 2023 +0100
19
Date:   Mon Jan 29 10:21:52 2024 +0530

• 4.10.0 was released around July, 2017;
• 4.11.0 was released around January, 2018 (around 7 months after the previous);
• 4.12.0 was released around March, 2019 (around 14 months after the previous);
• 4.13.0 was released around September, 2019 (around 6 months after the previous);
• 4.14.0 was released around May, 2020 (around 8 months after the previous);
• 4.15.0 was released around January, 2021 (around 8 months after the previous);
• 4.16.0 was released around November, 2021 (around 10 months after the previous);
• 4.17.0 was released around May, 2022 (around 6 months after the previous);
• 4.18.0 was released around March, 2023 (around 10 months after the previous);
• 4.19.0 was released around January, 2024 (around 10 months after the previous);

Looking at the data, it is clear that our current process is not adequate. Sometimes we barely release a major in a year and there is a case that we took more than a year to release a major. In my opinion, that occurs because the process we have is not well-defined and it seems we are not "caring" too much about it. What we are missing is planning and predictability; we do not know when the next release will be (after 4.20); we do not have a common goal. We delay months to deliver a release in order to fit a feature in a release. Thus, again, we lack planning. Regarding @rohityadavcloud question "who would enforce them, who would drive them?", well, it is the PMC's responsibility. Furthermore, we must comply with the Cyber Resilience Act (CRA).

About the disruptive changes, we can look at how other projects are doing it and adapt to our context. For instance, in one version, the GitLab community introduces features that are disabled by default and lets the users enable it; then, in another version (after validations and defining that the feature is stable), they enable it and remove the configuration. We can use a similar approach, like introducing new features/behaviors in a version and marking the previous behavior as deprecated, and then, in another version, we remove the previous behavior, as @DaanHoogland commented. The problem is: we do not have it clearly defined.

I believe we can benefit more from this discussion by separating it into small topics, for instance:

1. looking at our history, what would be the ideal cadence of major releases for our context and why?
2. how and when should we define the RM for each major?
3. how should we introduce disruptive changes and remove compatibility?
4. what are the community's common goals?
5. regarding minor releases, should we flexibilize it more or be more rigid?

[weizhouapache]

My 2 cents.

Regarding major releases, we do have plan to create 2 major releases (1 LTS, 1 non-LTS) a year. However, it appears nobody is interested in the non-LTS release. Now all recent major releases are LTS releases. We do not have adequate resources to maintain multiple LTS releases and main branch. IMHO it is ok to have major LTS releases every 8-10 months. All new features should be public on cwiki several weeks before code freeze on, for instance https://cwiki.apache.org/confluence/display/CLOUDSTACK/4.20+Design+Documents

We could create minor releases more often, for instance every 2-3 months. I have suggested before.

Regarding break changes, as many of you said, deprecate or announce in a LTS release, then remove or change in next LTS release.

1. looking at our history, what would be the ideal cadence of major releases for our context and why?
2. how and when should we define the RM for each major?
3. how should we introduce disruptive changes and remove compatibility?
4. what are the community's common goals?
5. regarding minor releases, should we flexibilize it more or be more rigid?

[DaanHoogland]

I agree with most of what @weizhouapache says here, except that I think that a new feature should be accepted by an issue or even an open PR as well. nit just the wiki

[rohityadavcloud]

"If it didn’t happen on the mailing list, it didn’t happen." https://theapacheway.com/on-list/

[GutoVeronezi]

And it is happening: https://lists.apache.org/thread/pohvscqqht43ftq3d4nsftpylg6tgxz0

[DaanHoogland]

I'm afraid @rohityadavcloud has a point here. The user mailing list is not the place it should happen, dev@ is where any decisions should be "coded". That doesn't diqualify this medium as a good discussion platform but anything has to be taken to dev@ for a decision. bit inconvenient truth. (pun very much intended)

[JoaoJandre]

Well, the original discussion was started on the dev mailing list... but it got no traction. That and @DaanHoogland's advice is what led me to create a discussion here. If you want to take this to the dev list, I see no problem with it, as long as we can keep the discussion going.

[GutoVeronezi]

Personally, I think it can reach more people here; however, I am more interested in having the discussion itself than where it is taking place. I am fine wherever we have the discussion phase.

[DaanHoogland]

@JoaoJandre I think discussing here is not bad. As long that if a vote is needed it happens on [email protected].