Please delete the new release and the insiders tag

[joaomoreno]

It makes no sense that it exists. I constantly keep removing it from my issues.

If we want such a functionality, simply tag it with the actual version number; although I suggest to not even do that since that is already in the issue's body.

cc @Microsoft/vscode for opinions

[jrieken]

although I suggest to not even do that since that is already in the issue's body.

Yes and it also makes us blind towards those issues that don't have any version number in them. They are equally good and valid issues and I don't like the extra highlight just because of the version number.

[bpasero]

Also get rid of the insider release tag please.

[Tyriar]

Moving to October as @chrmarti is on vacation

[chrmarti]

@Microsoft/vscode Anyone opposed to removing the new release and insiders labels?

Our initial impression was that insiders helps one identify regressions during endgame week and new release helps identify regressions after a release.

[bpasero]

no?

[kieferrm]

I share the sentiment that the new-release and insiders labels are not valuable if they are not limited to certain brief window during our iteration. Since we kept them longer than necessary they lost their meaning. However, the original motivation for introducing them was to address the following pain points:

• When we are right before a release and build insiders out of the release/x.x branches we need a way to quickly identify which issues have been newly filed against this insider build. A combined textual-structural search query is not good enough since it needs to be adapted to the latest build several times during the process.

• Right after we released a new stable build we are carefully tracking for regressions and issues to determine which issues should be considered for a recovery build. Since not all users upgrade to the build immediately (and we also have insiders) we get a mixture of issues across various builds.

• In both of the above cases, we need a predictably short response time for such issues and we need to be able to check if anything fell between the cracks.

Right now, trying to shepherd a build to stable release and not having any explicit indication/labels does not give me the right level of confidence that nothing is hiding without me seeing it.

Thus, I propose to bring back the labels strictly for the following time periods:

• The label insiders during endgame between us branching off master to release/x.x. We remove the label the very moment we release the new stable build. So, we typically would keep it from Friday - Wednesday.
• The label new-release for 4 days after we released a new stable build. So, we typically would keep it Thursday - Sunday.

[dbaeumer]

I would still not profit from such a label since I can't assume that the tagging is correct (there might be issues missing the label or issue having the label incorrectly). During the timeframe you described I go over my inbox carefully to decide which once need immediate attention. If there is one I set the candidate milestone. However having the labels will not cause any big harm either if I can simply ignore them and there is an automatic removal happening.

[alexdima]

@kieferrm

Right now, trying to shepherd a build to stable release and not having any explicit indication/labels does not give me the right level of confidence that nothing is hiding without me seeing it.

I share this sentiment, but I believe the root cause of this is the bot itself (automation), and it cannot be solved by adding more automation. i.e. No human forms anymore a high-level picture of how we are doing w.r.t. stabilization and regressions shortly before and shortly after a release.

Before the bot auto-assigning times, it used to be that the inbox tracker could answer with confidence if he/she has spotted any regressions or important issues while doing inbox triaging. The inbox tracker could, and often did, bring up individual issues or issue patterns in our daily standup, or would assign the important label, or would assign an issue to a milestone if a prompt investigation was in order. i.e. the inbox tracker could control what in other issue tracking systems is referred to as "severity".

All of this is lost by the automatic assigning of issues. For a certain issue X, if it is auto-assigned to an owner, no human other than the owner will be notified or otherwise read the issue. We don't ask of our inbox tracker role or of our endgame role that they include reading auto-assigned issues.

If a certain issue X reaches my inbox automatically, and if I am on vacation, or if I neglect to triage my personal inbox for a certain time, high severity issues might be sitting there and nobody will notice them. To make it worse, what if issue X belongs to a different owner and it got assigned to me due to "bot error"? If I don't triage my personal inbox in a timely manner, it might take a few days until the real owner of issue X gets to see it.

We celebrate the time gained by auto-assigning as a reduced time load on the inbox tracker, but that has two consequences:

• the one you identify that there is no confidence in how we are doing close before or close after a release,
• and the other one, that now everybody must be triaging their personal inbox daily. i.e. we have taken the time load and pressure from the inbox tracker and moved it to every team member.

I don't believe these two consequences were properly discussed before we decided to auto-assign issues.

Furthermore, as Dirk points out, more automation will not solve these two consequences. e.g. someone creating a valuable issue via GH and omitting to write a VSCode version number.

I suggest we disable bot auto-assigning during endgame and 4 days after a release. That will make it that a human, the inbox tracker, can tell us how we're doing.

[alexdima]

P.S. Can we also track and measure the error ratio of the bot.

e.g. for me in the past 24h: #35533 #35544 #35518 #35506

[chrmarti]

Remember that we have introduced the bot as a way to scale with the increased load in issue management. It is a simple fact that inbox tracking has become a full-time job without the bot and we should expect the volume of incoming issues to keep increasing with the growth of our user base.

The increased load for the inbox tracker also means it is harder for them to stay on top of what is going on and the increased pressure means quicker manual assignment and more errors in doing so. That are the same problems you criticize the bot for.

There are also advantages to issues being assigned immediately instead of a few times per day. Sometimes they can be dealt with much more quickly than when they sit in the inbox waiting for triage.

If the time before the bot seems brighter, it is because we didn't get as many issues filed back then.

[chrmarti]

Regarding #35533: Should the bot assign folding issues to @aeschli?
Regarding #35506: Should this be labeled editor-folding so the bot can "learn" from it?

[kieferrm]

@alexandrudima If the auto-assign bot is not good enough then disabling it during endgame and right after the release makes sense to me. However, I'd still think we should have the new-release and insiders labels in the time periods I describe above. Their whole point is to make it easier for the inbox trackers to focus their energy first on those issues that matter the most during in those specific days of our development cycle.

However, we also want to make sure the issue triaging bots we put in place are getting better. Otherwise, we won't be able to deal with the load we have.

[dbaeumer]

Side note: IMO the limiting factor with increasing load is not the inbox tracker. It is the component owner that has to deal with the issues.

@chrmarti can you describe how the two labels are assigned. As said having them makes only sense if I could focus on issues only having these labels. If I need to look at all issues anyways since I can't fully trust the labeling I will not win anything.

[alexdima]

@chrmarti @kieferrm

Remember that we have introduced the bot as a way to scale with the increased load in issue management. It is a simple fact that inbox tracking has become a full-time job without the bot and we should expect the volume of incoming issues to keep increasing with the growth of our user base.

However, we also want to make sure the issue triaging bots we put in place are getting better. Otherwise, we won't be able to deal with the load we have.

You touch on an important topic. The sheer amount of issues coming in is overwhelming and ... I definitely agree with that. I just don't share the opinion that an auto-assigning robot is the best (or even a good) way to deal with that.

IMHO an auto-assigning robot does not help in any way with the load on the team (viewed as a whole). An auto-assigning robot moves the load from the inbox tracker to the feature area owner. If a certain issue X has a certain time cost time(X), the auto-assigning robot does not reduce in any meaningful way time(X). It simply moves the time spent from the inbox tracker to the feature area owner.

Some human inbox trackers are nicer than others, and will not assign issues unless they can reproduce or if reproducing would be too hard. In this way, some nice inbox trackers would take the load and block an issue in the inbox until it could be reproduced or until sufficient information is available. I deeply appreciate such inbox trackers. The auto-assigning robot is the opposite of the nice inbox tracker.

What else could we do

IMHO, there are many better ways to reducing the issue load, both on the inbox tracker and on the team as a whole:

1. It does not make sense that we are a distributed team and we have a single inbox tracker at a certain time. Inbox tracking should always be done in pairs to take advantage of our time-zone differences and to reduce load on the individual inbox tracker.
2. Someone who is an inbox tracker should get a reduced load when planning an iteration, and during the endgame test assignment, etc.
3. Build bots that reduce time(X) instead of moving it to someone else in the team:
   3.a. a duplicate finder bot that suggests duplicates to the issue creator. If the match is over 90% or whatever the issue gets auto-closed as duplicate.
   3.b. an issue validator bot that closes issues in foreign languages, issues without a description, or issues without a version number if the text contains "bug" or "regression" i.e not a feature request.
4. Involve our community. One possible scheme could be: when an issue gets opened, the bot adds a label called "needs-triaging". We can ask members of the community to watch from time to time (if they want to help) the issues with "needs-triaging". Perhaps they can comment in a certain format. e.g. @bot question, @bot duplicate #issue, @bot feature request, and the bot can make use of our human friends. We can start with a select list of contributors that can drive the bot and learn from there.

All in all, I am not against the idea of a bot, I am against the current incarnation of the auto-assigning bot, which does not reduce my issue load, it increases it.

[Tyriar]

An auto-assigning robot moves the load from the inbox tracker to the feature area owner.

@alexandrudima this is where the issues should go anyway imo. It's a waste of time for the issue tracker to go through the process of searching for duplicates, asking for repro, following up and then assigning to the feature owner when the feature owner may just close it as a duplicate immediately anyway (because it's hard to search for issues when we have so many). I've seen this happen quite a lot with terminal stuff because I have an intimate knowledge of the issue and there are always 100+ open.

If we're talking purely about efficiency, someone with knowledge of the area will be able to resolve the issues with much less effort; inbox tracker: time(X), feature owner: time(cX) : c < 1 😉

I get that the editor has a lot of issues, my areas do as well. Some potential solutions:

• Get more people involved in the editor?
• Splitting up ownership of editor components more clearly so multiple people aren't looking at the same issues would help?
• Remove yourself from the editor label which you mention has a wrongly assigned several issues and add yourself to the sub-labels?

What else could we do

I like your suggestions, number 2 in particular I've already mentioned to @kieferrm and I try to keep in mind when we do planning.

---

We should not ignore the successes of the auto-assigner, for me it's turned inbox tracking basically from a full-time job for me to a couple of hours a day. For the terminal in particular the auto-assigner is pretty much always spot on because few other issues contain the word "terminal". Also the editor sub-labels seems to be doing a good job of assigning the right person automatically without any effort by the issue tracker.

[chrmarti]

I certainly agree we need to do more to make everyone's life easier. Having auto-labelling and auto-assignment for the inbox is only a first step that we can iterate on. I have talked with @rebornix and @Tyriar about the pain points with large individual inboxes and it is great that we are collecting more ideas on that here.

Our choice to improve the inbox tracker's job first really is a statement about what we thought can be improved without too much investment and not about what we would ideally want to achieve. E.g., duplicate detection and help in finding duplicates has been on our minds for a while, we just need to allocate a sufficiently large time slot to investigate and prototype our ideas for that.

Since I'm doing the inbox tracking this week I will check all incoming issues, including the automatically assigned ones, to make sure we keep an overview of all that is going on. We can discuss if that makes sense or if auto-assignment should be disabled. I will also enable the new release label, so we can collect some more experience with that and will remove it early next week, so it won't interfere with everyone's issue tracking too much.

[chrmarti]

Going through all incoming issues, including the auto-assigned ones, has worked well for me as the inbox tracker. It is interesting how little cues the classifier sometimes needs to correctly label an issue (from which the assignment is derived). What I was missing sometimes is a marker for at which issue I need to pick up reading again.

I have removed the new release label now. We can turn it back on for the recovery release.

@dbaeumer The new release label is added to issues matching the following with r being the version string configured in the main repo's .github/new_release.yml:

new RegExp(`VSCode Version:(.*[^\\d])?${r.replace('.', '\\.')}([^\\d]|$)`, 'i')

[dbaeumer]

@chrmarti thanks. I actually get a fair amount of issues without any release hint in the initial comment. The main problem I have with label right now is that I can't trust it and therefore ignore it.

[Tyriar]

The main problem I have with label right now is that I can't trust it and therefore ignore it.

@dbaeumer @chrmarti if there are particular labels with low accuracy we should disable them. Is the tasks label one of these?

[dbaeumer]

@Tyriar the task label works very well. I was referring to new release and insider label.

[V-ed]

4. Involve our community. One possible scheme could be: when an issue gets opened, the bot adds a label called "needs-triaging". We can ask members of the community to watch from time to time (if they want to help) the issues with "needs-triaging". Perhaps they can comment in a certain format. e.g. @bot question, @bot duplicate #issue, @bot feature request, and the bot can make use of our human friends. We can start with a select list of contributors that can drive the bot and learn from there.

This conversation is kinda old but I believe is still true today; often times I see issues that have alot of duplicates and while commenting about it seems to help, it would be one less task for the vscode team to deal with if there'd be some community interaction to help triaging issues.

Less than a month ago, a PR closed 5 issues, all of which were duplicates of the first...

I don't think giving the commnity full control would be the best idea - perhaps when commenting in the format that @alexandrudima gave as an example (@bot duplicate #[issue number]) the bot could add a label community-triaged, where the vscode team could know that these issues have already been analysed by the community and do the appropriate actions accordingly.
After a certain delay, if no response is made by the vscode team, apply a default action depending on what the comment was (example : @bot duplicate #1234 would close, after 3 days, the issue automatically if no further actions as been made).

I saw no mention of the point quoted at the start of my comment in this issue, maybe you discussed it in private, but I'd like to see the outcome of this idea as I'd like to help the team in a ay that could be potentially more helpful than commenting Duplicate of #1234 and waiting for a team member to view that comment before closing it. - Note that I don't mean commenting Duplicate of is not useful, but rather that such interactions could do more than simply commenting.

[chrmarti]

I have opened #56883 to continue the discussion on the community participating in triaging issues.