feat: Introduce a way to suppress violations

[softius]

Summary

Suppress existing violations, so that they are not being reported in subsequent runs. It allows developers to enable one or more lint rules and be notified only when new violations show up.

Related Issues

• Change Request: Introduce a system to suppress existing errors

[linux-foundation-easycla]

The committers listed above are authorized under a signed CLA.

• ✅ login: softius / name: Iacovos Constantinou (f9b3976, ed01be1, 4d73750, a99c456, c9a718c, 4462ce6, 485f684, 98779dc, dc2d940, b343ddb, 9e035f1, b8a1cf7, d5411af, 70a7c56, 42d8b95, ad5343d, bff622d, 0c3d9a4, 1a1cbba, 2d4dc84, 71ee661, 4fea00e, 46cf6ae, 48e9d0a, a94a50d, 861f1a4, 566e3b9)

[bradzacher]

the alternative we've introduced at Canva is that we designate specific rules as "in migration" and we only consider reports from those rules if they exist in changed files (according to git comparison against the main branch).

With this system developers must address lint errors if they touch a file but otherwise they can be ignored.

This does require integration with the relevant source control system - though we've found it works quite well.

[fasttime]

Thanks for the RFC @softius. Can you please sign the CLA as requested in this comment?

[nzakas]

Thanks for putting this together. It sound good in theory, but you haven't provided much in the way of implementation details. Please dig into the code a bit to see how this might be implemented.

[nzakas]

@octogonz Would really like your feedback on this based on your work on Rush Stack Bulk.

[octogonz]

The feature that we designed is part of @rushstack/eslint-patch. Kevin Yang summarized it in this blog post.

A key design question is how to keep bulk suppressions stable across Git merges, since they are stored separately from the associated source code. From a cursory glance, it sounds like this RFC is proposing to count the number of errors per file path. We prototyped that design, as well as a design that tracks line #'s, and also a design that tracks formatted messages ('x' is already defined.) which are more specific than rule names (no-redeclare). We also considered actually storing the suppressions in the source with a different comment syntax.

Ultimately we settled on a JSON file using a scopeId that loosely identifies regions of code (e.g. ".ExampleClass.example-method") while still being simple enough for humans to interpret without having to consult a specification. This notation is abstracted such that we can reuse the exact same scopeId syntax for tsc/tsconfig.json bulk suppressions (a related feature we're planning to release soon), even though it it is derived from the compiler AST instead of ESLint's AST.

There's probably still a lot that can be improved about our approach. However I can say that Microsoft and TikTok are using it successfully in two very large TypeScript monorepos, where style guide migrations involve hundreds of thousands of source files and megabytes of JSON. So it's worth studying even if you don't adopt it. 🙂

As far as future improvements, a feature I'd like to see is an API that enables the VS Code extension to highlight suppressed violations specially. Currently people either highlight them as normal errors (which I find confusing, since you have to run the CLI to find out whether you really need to fix it or not) or else don't highlight them at all.

@kevin-y-ang

[nzakas]

@octogonz thanks for the insights. I think we're past the point of adopting eslint-bulk wholesale, but would welcome your feedback on this RFC to see if there are any salient points that have been missed or problem areas you've encountered.

[octogonz]

thanks for the insights. I think we're past the point of adopting eslint-bulk wholesale, but would welcome your feedback on this RFC to see if there are any salient points that have been missed or problem areas you've encountered.

Agreed, the eslint-bulk tool was implemented as a monkey patch, therefore it can't be reused directly; the approach needs to be reworked in order to integrate this feature into official ESLint.

I've been really busy lately, but I'd like to help out with the RFC. Let me make some time to read through @softius's doc in more detail, then follow up with feedback.

[fasttime]

@softius just a reminder that there are some suggestions for you to apply. Please, let us know if you need any help.

[Humni]

I really support this RFC - we've been running our own baseline package (just a wrapper around eslint), which takes the same approach. We've been using this implementation on about 30+ prod projects with no complaints from our team. Hopefully, this implementation can assist with the RFC

[nzakas]

Apologies for the delay in getting back to this. Had a lot of post-v9 work.

I think there are some helpful ideas that eslint-bulk has that we should consider here:

1. Using the term "suppressions" instead of "baseline" - I think this clarifies what is actually happening. While people may not understand what generating a baseline is, most will understand when something is suppressed.
2. A way to indicate which rules should be suppressed. For instance, if I just want to suppress "no-eval", I should be able to do that. That may mean rethinking the command line flags. So maybe --suppress-rule <ruleId> and --suppress-all for all rules?

[nzakas]

@Humni that's great! Do you have any specific feedback on the implementation described in this proposal? How does it compare to what you're already using?

[kevin-y-ang]

Hi Nicholas, thanks for checking out the eslint-bulk project!

1. Using the term "suppressions" instead of "baseline" - I think this clarifies what is actually happening. While people may not understand what generating a baseline is, most will understand when something is suppressed.

Other names we considered were exemptions, pardons, and legacy-suppressions.

On a semi-related note, the "baselines" feature seems to very closely resemble the already existing eslint --max-warnings flag, which might be a source of confusion if the two features are both retained.

[nzakas]

On a semi-related note, the "baselines" feature seems to very closely resemble the already existing eslint --max-warnings flag, which might be a source of confusion if the two features are both retained.

@kevin-y-ang I'm not sure I follow. Can you explain what you mean by this?

[kevin-y-ang]

At a conceptual level, both the proposed baseline feature and the --max-warnings feature set a threshold of X errors where the system will alarm if there are >X errors.

At our company, we previously used eslint . --max-warnings X to block merge requests for packages with high warning counts. I believe this was the intended use case when it was originally created?

The message essentially being conveyed here is "Okay there are already X number of ESLint warnings in this package but we won't allow any more", while the message being conveyed with the baseline feature seems to be a more granular version of this: "Okay there are already X number of ESLint errors/warnings for rule R in file F but we won't allow any more."

Anyway, it's just a similarity I see, it's not a big deal.

[Humni]

@Humni that's great! Do you have any specific feedback on the implementation described in this proposal? How does it compare to what you're already using?

Probably the main decision to make for this RFC which style of error matching you use. These would be:

1. Exact Error Line Matching
2. Context Error Matching
3. Count with Context Matching (recommended)

With our current implementation, we use the Exact Error Line Matching, which does have some short falls. It works pretty well, the vast majority of PRs don't require any rework. There are cases where if you modify a line at the top of a file though, will cause all of the existing errors to be exposed. This approach was chosen due to simplicity. For the most part this does push a project to having a smaller and smaller baseline, however it does lead to developers touching the baseline "too much" due to them not wanting to fix errors not directly related to their PR.

We're currently exploring moving to a context aware error matching, so it only comes up if the nearest 3 lines of code are modified - LuminateOne/eslint-baseline#8. This would provide a solution to the issue above, but it's not yet tested but seems like a sensible approach (if it can be done in a performant way).

I would highly recommend an error count approach the same as PHPStan (as per comments from @ondrejmirtes), as that seems to sit with the development team better.

Only other question "feature request" I would add into this is to allow for "flushing" a baseline of excess/resolved errors only (and not add any new ones). Just a QOL improvement though as it's typically easy enough to see new lines added into the diff in the baseline.

[nzakas]

@Humni gotcha, that's helpful. So it sounds like the current RFC, which uses error counting, is what you'd prefer. 👍

Keep in mind, too, that we generally build things incrementally. Our goal here is to get a baseline (no pun intended) of support for suppressions such that we can continue to build on top of it as feedback comes in.

[nzakas]

@softius are you still working on this? There is some outstanding feedback to review.

[softius]

At a conceptual level, both the proposed baseline feature and the --max-warnings feature set a threshold of X errors where the system will alarm if there are >X errors.

At our company, we previously used eslint . --max-warnings X to block merge requests for packages with high warning counts. I believe this was the intended use case when it was originally created?

The message essentially being conveyed here is "Okay there are already X number of ESLint warnings in this package but we won't allow any more", while the message being conveyed with the baseline feature seems to be a more granular version of this: "Okay there are already X number of ESLint errors/warnings for rule R in file F but we won't allow any more."

Anyway, it's just a similarity I see, it's not a big deal.

@kevin-y-ang the baseline doesn't take warnings into consideration or alter warnings in any way. So the message conveyed is ""Okay there are already X number of ESLint errors for rule R in file F but we won't allow any more.""

[nzakas]

The same command can be used afterwards when new errors are added in .eslintrc, but also to remove any unmatched rules.

Right, and this was my concern -- it feels like --baseline is doing a lot that isn't apparent to users. Specifically, I just don't think the word "baseline" correctly indicates any of what it's doing vs. explicit flags.

To clear unmatched rules you need to execute eslint suppress prune - which means that we will need one more extra flag i.e. --prune-suppressions if we decide to go down that path.

I think this is preferable because it clearly indicates what is happening.

[softius]

@fasttime @nzakas I updated the terminology and concept from baseline to suppressions. The new options are now documented for the ESLint CLI. I also separated the execution details from the CLI options for clarity. Implementation notes have been updated too.

Having a closer look at the source code, I have noticed that it messages suppressed with eslint-disable. Should we handle messages suppressed by this RFC the same way ?

[nzakas]

This is looking really good. Just a few minor points to address.

[softius]

Minor points addressed.

[nzakas]

Thanks for all your work on this. I'm pretty happy with the current state, so marking as approved.

We still need other TSC members to approve to move forward.

[fasttime]

Thanks for the update. Maybe I'm misunderstanding something, so bear with me. It would be helpful if you could add a draft implementation to show how this function will work.

To clarify, this is an example I would like to understand. Suppose you have a file test.js where the no-undef rule would normally report three errors:

/* eslint no-undef: "error" */

a = b + c;

Playground link

What will the function return when the suppression file specifies a count of 2? This is what the arguments will look like:

const results = [{
    filePath: '/my-project/test.js',
    messages: [{
        ruleId: 'no-undef',
        severity: 2,
        message: "'a' is not defined.",
        line: 3,
        column: 1,
        nodeType: 'Identifier',
        messageId: 'undef',
        endLine: 3,
        endColumn: 2
    },
    {
        ruleId: 'no-undef',
        severity: 2,
        message: "'b' is not defined.",
        line: 3,
        column: 5,
        nodeType: 'Identifier',
        messageId: 'undef',
        endLine: 3,
        endColumn: 6
    },
    {
        ruleId: 'no-undef',
        severity: 2,
        message: "'c' is not defined.",
        line: 3,
        column: 9,
        nodeType: 'Identifier',
        messageId: 'undef',
        endLine: 3,
        endColumn: 10
    }],
    suppressedMessages: [],
    errorCount: 3,
    fatalErrorCount: 0,
    warningCount: 0,
    fixableErrorCount: 0,
    fixableWarningCount: 0,
    source: '/* eslint no-undef: "error" */\n\na = b + c;\n',
    usedDeprecatedRules: []
}];

const suppressions = {
    "test.js": {
        "no-undef": {
            count: 2
        }
    }
};

You can see that the input results contains details about each error, while the suppressions are just numbers. What will the call to applySuppressions(results, suppressions) return in this case?

[softius]

In that case it will return an updated copy of results and an empty object for unmatched (because all suppressions are matched).

// results
const results = [{
    filePath: '/my-project/test.js',
    messages: [
    {
        ruleId: 'no-undef',
        severity: 2,
        message: "'c' is not defined.",
        line: 3,
        column: 9,
        nodeType: 'Identifier',
        messageId: 'undef',
        endLine: 3,
        endColumn: 10
    }],
    suppressedMessages: [],
    errorCount: 1,
    fatalErrorCount: 0,
    warningCount: 0,
    fixableErrorCount: 0,
    fixableWarningCount: 0,
    source: '/* eslint no-undef: "error" */\n\na = b + c;\n',
    usedDeprecatedRules: []
}];

// unmatched
{}

[fasttime]

If I understand correctly, this means removing the first count messages with a particular rule-id from each result, where count is the suppression count. So, in the above example a and b would be suppressed but not c, because in the input result, a and b are sorted first in order of location.

I think this behavior would be very surprising to users. This means that some rule violations would be reported only for (the bottom) part of a file.

I think it would be better to report all messages unfiltered if the suppression count is less than the number of messages, because in that situation we can't make a meaningful selection. So for each file and rule-id we should always report either all messages, or none of them.

[nzakas]

@eslint/eslint-tsc looking for another approval on this.

[mdjermanovic]

Suggested change

	A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this is option can accept an array of string values.
	A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this option can accept an array of string values.

[mdjermanovic]

Suggested change

	eslint --suppress-rule '@typescript-eslint/no-explicit-any' --suppress-rul '@typescript-eslint/member-ordering' ./src
	eslint --suppress-rule '@typescript-eslint/no-explicit-any' --suppress-rule '@typescript-eslint/member-ordering' ./src

[mdjermanovic]

In this case, will there be an error message, and what would it look like? Technically, will it be a lint message passed to the formatter along with other lint messages for the file, or a separate output?

[mdjermanovic]

Having a closer look at the source code, I have noticed that it messages suppressed with eslint-disable. Should we handle messages suppressed by this RFC the same way ?

That's a good question. I think it makes sense to insert lint messages suppressed by this feature into LintResult#suppressedMessages before passing lint results to the formatter. @nzakas @fasttime thoughts?

[nzakas]

I think it makes sense to insert lint messages suppressed by this feature into LintResult#suppressedMessages before passing lint results to the formatter

I agree. Makes sense. 👍

[fasttime]

I think it makes sense to insert lint messages suppressed by this feature into LintResult#suppressedMessages before passing lint results to the formatter.

Makes sense. I think that errors suppressed by the suppressions file will have no suppressions array attached to them, so tools should be able to tell them apart from errors suppressed per eslint-disable directive because of that.