Suggested improvements to the incubating/graduated levels

[carolynvs]

TAG Contributor Strategy is spinning up a short-term group to collect feedback on CNCF project levels and process for completing the incubating and graduated levels. If you have feedback that you would like to share with us before we get started, we welcome you to leave it here on this issue. We are looking for feedback (positive and negative), particularly from projects that have attempted to move up a level.

So if you have ideas for how the process or level requirements could be improved, please leave a comment!

[AlexsJones]

Great initiative.

Regarding the endeavour of collecting feedback;

I feel strongly that generating quantitative data here is the key to derive impartiality on improvements and side-step some of the more emotive encounters people may have had.

Having been through the levelling process as an advisor on a few projects now, I believe there is often suspicion and frustration when there are delays or rejections from a project application.

Therefore, I would advise we generate a set of initial hypothesis and perhaps Likert style questionnaire with supporting interviews for context. The net benefit would be a transparent scientific process to identify challenges in the current TOC assessment process and to gain credibility into the validity of our own enquiries.

Needless to say, happy to help if it's useful.

[TheFoxAtWork]

Additional related issues on the TOC repo with existing discussion that may be beneficial for this group:

• Clarify Process of Projects with Sub Projects toc#1015
• Introduce framework of maturity levels for different assets within a project organization toc#1009
• Add clarification to CNCF graduation criteria toc#992
• Add a formal Govenance review to Graduating DD toc#963
• Community stability for maturing projects - updates to Graduation toc#907
• Clarify graduation submission requirements/review toc#1013

[xmulligan]

I'm also happy to help on this once it kicks off

[divya-mohan0209]

I'm happy to help on this too!

[kaiwalyakoparkar]

I would like to help too :)

[thedevelopnik]

I'm interested in helping on this.

[craigbox]

I would be happy to lead, or at the least contribute to, this effort.

I definitely don't presuppose an outcome but I would like, for example, redefining the idea of levels entirely to be on the table. This question is larger than issues related to contribution; it also relates to how companies and projects market themselves based on levels, and the incentives for them to want to move between them.

[TheFoxAtWork]

Adding some notes here as a placeholder for these discussions when they happen:

• Licensing alignment is the the responsibility of GB and CNCF Staff to resolve (as the TOC are not lawyers). TOC checks on due diligence for licensing alignment are to confirm that it has been done or that it is in progress (tracked via ticket).
  • Additional alignment checks that fall outside the scope of technical viability and maturity for moving levels or due diligence should be carefully considered as to whether they are blocking (must be done) or tracked (must be in progress) with documented understanding as to an unexpected outcome of in progress. For instance, if a licensing exception was requested at time of application to move levels but was not approved until after the project moved levels - what then?
• Existing moving levels related docs that touch on these topics and will need updated:
  • Our DD template on the repo dictates documentation of alignment with CNCF where license is called out.
  • The Due Diligence process calls out that licensing “might” inform answers to TOC questions of which there are 4, one is the point on alignment however it is ONLY for “CNCF values and mission” and states that this is handled by CNCF Staff.
  • The Project onboarding calls out that the CNCF will do or help the project to “adopt a license scanning tool” placing this responsibility on the CNCF.

[duglin]

Sorry for not adding this sooner...happy to help as well.
What the status of this work?

[TheFoxAtWork]

I realize we didn't include this on the issue:
For folks interested in this, the work is planned to kick off in Aug/Sept to allow for time to collect enough data sources for this group to consider in their recommendations/plan.

[duglin]

Is there a list that we can look at of the data sources gathered so far?

[TheFoxAtWork]

Not precisely - here are the related messages from the TOC mailing list: Project Leveling Community Feedback Task Force & Moving levels discussion next steps. Here is the YouTube link to the recorded discussion that prompted this issue. The identified initial set of sources are:

• Maintainer Survey (in progress)
• Staff interviews and discussions with projects at KCCN EU
• TOC member discussions with projects during DD and moving levels (We're asking projects to record their thoughts on this issue, but if they are uncomfortable doing so, we're noting them and will be posting here)
• TOC specific observations/experience with projects (to be captured)

This group would also be identifying other sources of data. Many of the issues linked above in the comments already capture some experience/observations from community members, TAGs, and maintainers, however we also want to understand if there are measurable observations about moving levels (project readiness, project status, etc.) to reduce the workload on stakeholders.

[TheFoxAtWork]

Additional item - follow up to establish Process for transitioning Google Docs in CNCF-based workspace formally as part of moving levels.

[TheFoxAtWork]

Additional item adding here for situational awareness - (this is on TOC to do) clarify the "how" in conducting DD reviews and adopter interviews to ensure consistent experiences for projects moving levels. Adopter interview lists must cover more than 3 adopters (recommend 5-7 across multiple industry verticals like finance, health, telco, etc.) to request from in order to ensure we have enough availability and do not stall the process on an interview that may never be scheduled.

[duglin]

Another item related to this: Clarify graduation submission requirements/review

[TheFoxAtWork]

thank you! adding this above in the list. (edit) link: #366 (comment)

[amye]

Updates: TOC has worked on defining scope + a rough timeline.

Next steps: Interested parties for the committee should comment here!
We'll be looking to set up a kickoff meeting for the week of August 21st.

Timeline

Completion of scoping documentation: August 1st
Committee formation + public comment: August 1st-18th
Kickoff: week of August 21st
Midpoint: week of September 18
Submission of recommendations to TOC: October 23rd
Public comment opens: week of October 30
Adoption by TOC: week of November 27th

Background

The maturation process (commonly referred to as moving levels) is a process by which the CNCF, through community driven efforts, can consistently express a given project’s alignment with their journey to cross the Chasm. It is intended to convey to potential adopters, the market, End Users, and industry a project’s ability to meet or exceed expectations if it were to be integrated or used by adopters. A definition of adopters can be found on the TOC repo in the FAQs. Effectively, the maturation level or state has enough detail to inform Adopters of what they are getting and can plan integration and reliance accordingly.

The Technical Oversight Committee (TOC) is the technical governing body of the CNCF and consists of elected technical experts in their field. They are responsible for facilitating neutral consensus of the technical vision of the CNCF. As a part of this work, they oversee and admit projects into the foundation through the maturation process.

As the foundation has evolved and cloud native projects approach or reach market commoditization (such as Kubernetes), new considerations and concerns have been brought forward that identify gaps and overlaps in the current maturation process. It is imperative that the TOC, through project and community feedback and recommendations, iteratively evolve the maturation process to align with current market conditions, adopter needs, and increase the transparency of process execution for the benefit of CNCF Members, projects, community adopters, and open source as a whole.

Scope:

This working group will be performing a data analysis based on community feedback regarding experiences for moving levels. Based on the results of their analysis and the objectives outlined below, the working group will formulate recommendations to adjust, change, or restructure how the TOC determines the technical maturity and viability of CNCF projects at all lifecycle stages and growth states which occur in the Chasm model. Everything will be considered as recommendations to the TOC for final determination.

Lifecycle stages of the Chasm Model:
https://www.cncf.io/wp-content/uploads/2023/06/crossing-the-chasm-1800x687.png

In scope
Defining levels of maturity within the lifecycle
Defining criteria and common characteristics of each level of maturity
Defining archival as a state of growth with criteria and common characteristics to indicate this state of growth
Defining supplemental as a state of growth (projects that will not mature, but are supplemental to the ecosystem by adopters) with criteria and common characteristics to indicate this state of growth
Restructuring of the process by which a level and state evaluation occurs
Defining the stakeholders and key roles responsible for executing the process at each step
Defining what, if any, additional attributes can be recorded or captured to better express domain-specific maturity characteristics and the corresponding evaluation by which an attribute(s) could be expressed upon completion.

Out of scope
Defining CNCF benefits to projects at levels or states
Identifying CNCF staff specific roles, functions, activities as part of the moving levels process
The moving levels process cannot include elements that require CNCF Staff to execute. It is expected and anticipated Staff provide support to ancillary activities that input to or result from the moving levels (such as announcements, coordination, licensing scans)

Objectives:

The deliverables of this working group should focus on meeting the following foundational objectives:

• Streamlining the overall moving levels process to increase the speed by which applications, evaluation, and movement to vote may occur
• Provide specific and concrete criteria as core to the eligibility to move levels (blocking)
• Identify principles, indicators, and other areas that weigh or influence validation of a project’s assertion of alignment with the level for which they apply
• These may be considered blocking when taken in aggregate and are likely to be heavily driven by the specific project, how it executes in those areas for what it is designed to accomplish, how it is meant to be used, etc.
• Recommendations align with and cannot be in conflict with the TOC principles as published
• Additional principles may be suggested, such as any focused on the security of CNCF projects (already in consideration for an update due KCCN NA)
• Include considerations impactful for an Adopter’s decision to use a project
• Increase the amount of data and observations that may be collected or ascertained programmatically about projects with meaningful value to their maturity
• Consider both quantitative and qualitative elements that describe, express, or validate a project’s assertion of alignment with the level for which they apply
• Identify clear points of check-in with projects while they exist and function within a level or state to provide support and assistance in maintaining their viability and encouragement to become eligible for applying to the next level

Recommended Deliverables of the group

The following list is a recommendation of deliverables the group may provide. It is expected the group will provide additional deliverables beyond this list. Should a recommended deliverable not be included in the final work product of this group, a reasonable justification should be provided to transparently convey its absence to the ecosystem.

• Templates for consistently expressing eligibility, alignment, and other data necessary to move to a vote
• Project specific guidance on expectations, process steps, and supplemental material to assist in reaching eligibility and asserting alignment with a level or state
• Clearly documented workflow that structures the entirety of the moving levels process

[jberkus]

Streamlining the overall moving levels process to increase the speed by which applications, evaluation, and movement to vote may occur

I think we should add:

• Reducing the time and effort requirement for project leaders, TAG and TOC members, and staff

... since that's a big one, and where I'm hearing the most complaints now. It's not so much the amount of calendar time as the amount of people-hours.

Also, given current TOC activity, we should probably add:

• Make recommendations to the TOC around what parts of maturity evaluation could be delegated to TAGs, and how that process could work.

[angellk]

Thanks @amye - I am interested in participating.

[duglin]

@amye please add me to the list

Now some rambling...

The list of objectives/scope are focused on two main things (to me):
1 - how to speed up the graduation process
2 - the process is there to help the community know the state/maturity of projects. For example:

It is intended to convey to potential adopters, the market, End Users, and industry a project’s ability to meet or exceed expectations if it were to be integrated or used by adopters.

I'd like to suggest (and maybe this is implied) that part of what this committee should examine is the purpose of the levels. For example, if a graduated project is "mature" and the CNCF is giving it's seal-of-approval... today, what about a year from now? What if the project still technically meets all of the criteria to remain an active CNCF project but some of the subjective aspects of the TOC's vote to move it to "graduated" status no longer apply? Should they be moved back to "incubator"? If not, why not? Doesn't allowing it to remain hurt the meaning of having the "graduated" label? If a project can not be downgraded (just archived) then I believe that we should change the criteria for being promoted since retaining the promotion status isn't based on that same criteria as being promoted, and we're then holding existing graduated projects to a lower standard than upcoming ones.

I'd actually prefer something more along the lines of a boolean decision. Either a project meets our criteria to be in the CNCF or not (meaning no levels at all). Factors like age of the project, or subjective measurements shouldn't be part of the decision. For example, is the project related to CloudNative tech? Does it have a community rather than a single vendor driving it's future design? (notice it's ok to be a single vendor if it's in maintenance mode, but not if it's doing major design changes - in which case it should be examined (regularly) to see if being single vendor is ok for their situation). Stuff like that.

Oh I should say, this would also greatly speed up the overall process and reduce the workload on the TOC, IMO

Looking forward to the chats!

[craigbox]

A wholehearted +1 to Doug's thoughts: they are far from rambling (and far from new).

The charter empowers the TOC in "creating a conceptual architecture for the projects". I read this as the entire model of Sandbox, Incubating and Graduation is within the scope of the TOC to change. (Indeed, it's worthwhile remembering that Sandbox started out as "Inception", and that Doug was there at the time.)

In terms of the scope put forward, how much is the Chasm model open for debate? As Cloud Native as a concept has probably crossed the chasm, I would suggest that incubating projects are commonly in use by industry pragmatists, and waiting for projects to be graduated to use them is a quite a skeptical take. (It is also, as Doug points out, "one way".)

Changes to how the TOC categorises projects could also have necessary impact on what the staff do. For example, if there were no levels, then the way that promotion was done for projects would necessarily have to change.

Please count me in for the WG.

[itaysk]

I would love to participate if possible. I have some preexisting thoughts and feedback based on personal experience and community discussions

[lizrice]

I'd be interested in participating in the WG if time allows; I have a lot of opinions on this from experience on both sides of the process. Unfortunately I've got a conflict with the TOC meeting that's discussing this today, so I'll have to just raise a few of my questions and thoughts here.

Defining supplemental as a state of growth (projects that will not mature, but are supplemental to the ecosystem by adopters) with criteria and common characteristics to indicate this state of growth

I'm not sure I understand the phrase "supplemental to the ecosystem by adopters". Is this proposed to be another phase in the maturity model? If so, let's be very clear right from the start what benefits a project in this phase has, and how end users are supposed to interpret this phase. (One of the mistakes we made with Sandbox was that initially projects were getting marketing benefits, which wasn't in line with what the TOC had really intended, and led to an overwhelming influx of projects.)

we're then holding existing graduated projects to a lower standard than upcoming ones.

This is indeed the case today

To echo what @duglin and @craigbox said, I think it's time to step back and review the overall maturity model. For example, with the benefit of hindsight I now believe that opening Sandbox up to a wide range of projects was a mistake (one that I will happily take my share of responsibility for as I was TOC Chair in its early stages). We always expected Sandbox to be experimental, and for some projects to fail, but I don't think we properly understood the consequences of taking the IP from the project's creators, and how that affects the motivations of different stakeholders.

One last thing: the first of the Principles in the CNCF Charter is that Fast is better than slow. The foundation enables projects to progress at high velocity to support aggressive adoption by users.. Let's keep that front and center, and use that as a measure of how successful the process is. Something we lost along the way is the idea that the TOC community feels motivated to nurture projects along the path to graduation, and feels good about it when they get there, and even better when they get there quickly.

[TheFoxAtWork]

Quick update from today's TOC call to discuss this (link to recording coming soon from @amye ):

• this work will be broken up into two areas: immediate improvements that can be recommended in the existing timeline, and longer term changes/considerations of the levels to be initiated in early to mid spring 2024.
  • This was decided due to the complexity and depth of discussion needed to consider longer, more impactful improvements overall within the suggested timeframe. We need improvements now, these may be short-term to provide immediate improvement and will allow for reconsideration in the spring as input to the larger discussion.
• Duffie will be the primary TOC member supporting this activity, with additional TOC members (myself, Matt, Dave, Cathy) providing additional support
• Much of this initial work is not going to be meeting heavy, rather specific improvements and recommendations will be driven from the information collected on this issue, as well as other issues referenced here or in the TOC repo.

Please let me know if i missed anything.

[amye]

Recording is here:
https://www.youtube.com/watch?v=Sj36MUY20m4

[craigbox]

Thanks Emily & Amye; I've caught up on the meeting. I would like to suggest running the two areas as parallel streams, starting both ASAP.

I assert the following things are true:

• it is considered to be required to be a CNCF project to be successful in the Cloud Native ecosystem today
  • projects list "Find us on the CNCF Landscape" on their README to suggest at affiliation
  • I have even seen the phrasing "Project is a CNCF Landscape Level Project"
  • see also: Istio
• as of last week, it could well become a requirement to be in an open source foundation to be successful in open source in general
• the number of project submissions is already growing ~exponentially while the number of TOC members has remained effectively constant
• the wait time for allocation of a TOC member to do a due diligence is upwards of 6 months

All of that suggests that the CNCF could conceivably suffer congestion collapse before Spring 2024.

I also feel compelled to assert the "Fast is better than slow" principle here. You have countless years of experience in the CNCF and open source in just Liz, Doug and myself, all of whom were all ready to help months ago, but were told "wait until August", and now we're being asked to wait another 8 months. ¹

Footnotes

1. I had to do the math because my springs and summers are different to yours! ↩

[amye]

I assert the following things are true:

• it is considered to be required to be a CNCF project to be successful in the Cloud Native ecosystem today

  • projects list "Find us on the CNCF Landscape" on their README to suggest at affiliation
  • I have even seen the phrasing "Project is a CNCF Landscape Level Project"

Anyone can contribute to the Landscape, that's unrelated to this work.

This work is evaluating and suggesting improvements to projects moving levels, not projects being included at a sandbox level. That work was Q3 2022, resulting in a new sandbox process for 2023.

[amye]

Here's the list of people who have volunteered on this issue:
@AlexsJones @xmulligan @divya-mohan0209 @kaiwalyakoparkar @thedevelopnik @craigbox @duglin @angellk @itaysk @lizrice

We'll be closing committee interest EOD Pacific on Friday, 8/18 with a kickoff to come the week of August 23rd.
The intention is that the meetings will be minimal and most work will happen asynchronously.

[jberkus]

We'll need a TAG-CS rep on the commitee as well. I'll let you know soon who that is.

[TheFoxAtWork]

the number of project submissions is already growing ~exponentially while the number of TOC members has remained effectively constant

Yes, the volume of applications is growing, and anticipated to continue growing at an exponential rate, very likely in part to to the items you mentioned (and probably several others). The number of TOC members is constant. There are a few challenges in altering that, and while I don't believe it is out of possibility, it is a discussion for another space (GB).

That being said, here are a few points to consider regarding throughput and capacity:

• Quorum allows us to move forward when members are absent. Coordination among our current members, representing multiple organizations, industries, and timezones is tricky. Attempting to reach quorum with more individuals is difficult. We need the right balance of expertise, information, time, and people to be efficient and reach the velocity our projects desire.
• Increasing the pool of members could allow us to cover more areas or increase our throughput (within limits), but also slows the pace at which we can respond or take on work. Its double-edged.
• The current structure and supporting process needs evolution. To me, our processes need an automation transformation to provide the TOC with meaningful, consistent, and relevant information so that we spend less time looking for the information and more time connecting with projects for faster outcomes.

the wait time for allocation of a TOC member to do a due diligence is upwards of 6 months

Yes, it is/was. In the past 6 months however, the current TOC has begun moving faster on projects seeking to move levels. We're still constrained by the availability of our members and other key stakeholders in this process (this is not likely to ever change), however we are improving our own methods and means of due diligence to expedite it without compromising the quality of the work. I anticipate the TOC will continue to reduce our backlog, while we continue to balance other community and project challenges and evolve the process alongside the community.

I also feel compelled to assert the "Fast is better than slow" principle here.

Agreed, that is the primary reason for the smaller iterative approach, it is more likely to be completed sooner than a more in-depth refactor of the entirety of our levels. We also want to ensure we're taking an informed approach, trying new things, seeing what works and doesn't, adjusting, and iterating.

You have countless years of experience in the CNCF and open source in just Liz, Doug and myself, all of whom were all ready to help months ago, but were told "wait until August", and now we're being asked to wait another 8 months.

You all are valued community members whose experience is appreciated and often sought out. I understand your frustration at being asked to wait but the invitation to participate in this group, with the more iterative focus, as well as the next is still open. Your fervor and passion on this topic and different approach would be appreciated in both groups.

For myself, I see opportunities such as this as a way to experiment and to inform a longer term strategy that solves not only today's problems, but tomorrow's as well. Ideally in a manner that sets us up to address whatever comes in the years to follow.

[TheFoxAtWork]

TOC now has two milestones to reflect the distribution of this work. The first: https://github.com/cncf/toc/milestone/1. and the second: https://github.com/cncf/toc/milestone/2

[amye]

Invites are out for a kickoff meeting next Wednesday, August 23rd.

[jberkus]

TAG-CS's representatives will be myself, and @Riaankl

[craigbox]

That's two participants in NZ now. Is it possible to have a meeting schedule that includes our waking hours?

[divya-mohan0209]

+1 to what @craigbox said. The current meeting time is late evening & doable for me, personally. However considering there's participation across the globe, would it be feasible for us to have two meetings like we do for CNCF Ambassadors?

• Meeting number 1: EST (morning), CEST(late afternoon), and APAC (late evenings)
• Meeting number 2: ANZ (morning), PST (afternoon-ish), EST (evening-ish)

Or we could alternate between these two options for every meeting?

[amye]

Updated Scope, as of 8/23:
Clarifying criteria and common characteristics of each requirements for review for the defined existing maturity
Identifying process improvements connected to the criteria

• Criteria to move levels
• clarity of the “asks” for the review is part of what we should focus on
• rigorous definitions that could be self-assessed
• Outcome oriented, focus on the intent of the requirements to allow projects to reach desired outcomes in a way that suits the project.
• Clearly spell out which things are optionally recommended, and which are required
• Balancing objective criteria with subjective to give projects more guidance

We'll be actively working in #project-criteria-task-force to provide recommendations ~early October. I'll provide more updated timelines as we have them.

[craigbox]

FYI, the recommendations have been merged into the TOC repo.

[craigbox]

PTAL at my experience actually following the process.