x/tools/build/dashboard: email notify on subrepo breakages

[adg]

Send mail when a sub-repository fails to build.

[rsc]

Comment 1:

[The time for maybe has passed.]

[rsc]

Comment 2:

Labels changed: added go1.3maybe.

[rsc]

Comment 3:

Labels changed: added release-none, removed go1.3maybe.

[rsc]

Comment 4:

Labels changed: added repo-tools.

[odeke-em]

Kindly looping in @dmitshur @andybons @cagedmantis @toothrot.

Could we perhaps get subrepo failures to report to a public mailing list that folks can subscribe to, perhaps golang-builds, and CC the CL authors? It’s been 8 years and we’ve lived without this functionality, but would be nice to auto report these failures indeed, as Andrew Gerrand suggested, instead of folks having to manually look at the build screens. Or does such functionality already exist? Thank you.

[toothrot]

This functionality does not exist. I agree that build breakage notifications are a good idea. Those of us working on the build infrastructure are currently focusing on stability and reliability of releases, which definitely feels related, but I do not know when we will be able to commit to taking on this specifically.

[ianlancetaylor]

Just a note that a problem with e-mail based notifications is that it's very easy for them to trigger too often. People then routinely ignore them, and they serve no useful purpose. So if we do try to implement this, it's essential to minimize the number of messages, and to only send them when we are really certain that something has gone really wrong, and that we aren't just seeing flakiness.

Just as an example, perhaps it would be appropriate to send an e-mail message if a build fails five times in a row and there is some similarity in the output log for the failing builds.

[bcmills]

In the absence of automated alerts, checking the subrepo build status should probably be part of some kind of periodic triage rotation, so that we don't build up a large number of regressions unnoticed during the release cycle.

@golang/release, is there an existing rotation checklist that this could be added to?

[bcmills]

(For a recent example, #48078 during the current cycle appears to have gone unnoticed for around two weeks. It's a fairly obscure regression, so it isn't surprising that it wouldn't be noticed unless someone is actively keeping an eye on the dashboard.)

[toothrot]

@bcmills The x/repos are checked before cutting a release, but not as part of the release rotation itself, which is already stretched fairly thin.

I think checking before a release is important for our team to do as a last-chance sanity check before we cut a release. We typically start paying closer attention a few days before we know a release is coming.

For x-repos, I still believe that it is first and foremost the responsibility of the team or individual that owns the repo to be responsible for its health. The project is simply too big for us to be capable of watching everything every day. I don't think we should add more to the release rotation to make up for that, but we can guarantee that we won't release something that we know is broken.

I think the best solution here is improved tooling for detecting regressions, which I am sure is an uncontroversial thing to say, but it is unclear when we may be able to provide it.