-
-
Notifications
You must be signed in to change notification settings - Fork 66
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Introduce a way to suppress violations #119
base: main
Are you sure you want to change the base?
Conversation
|
||
Furthermore, we are adding one more reason to exit with an error code (see "Maintaining a lean baseline"). This might have some negative side-effects to wrapper scripts that assume that error messages are available when that happens. We could introduce a different exit code, to differentiate between exiting due to unresolved errors or ignored errors that do not occur anymore. | ||
|
||
## Alternatives |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the alternative we've introduced at Canva is that we designate specific rules as "in migration" and we only consider reports from those rules if they exist in changed files (according to git
comparison against the main branch).
With this system developers must address lint errors if they touch a file but otherwise they can be ignored.
This does require integration with the relevant source control system - though we've found it works quite well.
Thanks for the RFC @softius. Can you please sign the CLA as requested in this comment? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for putting this together. It sound good in theory, but you haven't provided much in the way of implementation details. Please dig into the code a bit to see how this might be implemented.
@octogonz Would really like your feedback on this based on your work on Rush Stack Bulk. |
The feature that we designed is part of @rushstack/eslint-patch. Kevin Yang summarized it in this blog post. A key design question is how to keep bulk suppressions stable across Git merges, since they are stored separately from the associated source code. From a cursory glance, it sounds like this RFC is proposing to count the number of errors per file path. We prototyped that design, as well as a design that tracks line #'s, and also a design that tracks formatted messages ( Ultimately we settled on a JSON file using a There's probably still a lot that can be improved about our approach. However I can say that Microsoft and TikTok are using it successfully in two very large TypeScript monorepos, where style guide migrations involve hundreds of thousands of source files and megabytes of JSON. So it's worth studying even if you don't adopt it. 🙂 As far as future improvements, a feature I'd like to see is an API that enables the VS Code extension to highlight suppressed violations specially. Currently people either highlight them as normal errors (which I find confusing, since you have to run the CLI to find out whether you really need to fix it or not) or else don't highlight them at all. |
@octogonz thanks for the insights. I think we're past the point of adopting eslint-bulk wholesale, but would welcome your feedback on this RFC to see if there are any salient points that have been missed or problem areas you've encountered. |
Agreed, the I've been really busy lately, but I'd like to help out with the RFC. Let me make some time to read through @softius's doc in more detail, then follow up with feedback. |
@softius just a reminder that there are some suggestions for you to apply. Please, let us know if you need any help. |
I really support this RFC - we've been running our own baseline package (just a wrapper around eslint), which takes the same approach. We've been using this implementation on about 30+ prod projects with no complaints from our team. Hopefully, this implementation can assist with the RFC |
Co-authored-by: Nicholas C. Zakas <[email protected]>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Apologies for the delay in getting back to this. Had a lot of post-v9 work.
I think there are some helpful ideas that eslint-bulk has that we should consider here:
- Using the term "suppressions" instead of "baseline" - I think this clarifies what is actually happening. While people may not understand what generating a baseline is, most will understand when something is suppressed.
- A way to indicate which rules should be suppressed. For instance, if I just want to suppress "no-eval", I should be able to do that. That may mean rethinking the command line flags. So maybe
--suppress-rule <ruleId>
and--suppress-all
for all rules?
@Humni that's great! Do you have any specific feedback on the implementation described in this proposal? How does it compare to what you're already using? |
Hi Nicholas, thanks for checking out the eslint-bulk project!
Other names we considered were exemptions, pardons, and legacy-suppressions. On a semi-related note, the "baselines" feature seems to very closely resemble the already existing eslint --max-warnings flag, which might be a source of confusion if the two features are both retained. |
@kevin-y-ang I'm not sure I follow. Can you explain what you mean by this? |
At a conceptual level, both the proposed baseline feature and the --max-warnings feature set a threshold of X errors where the system will alarm if there are >X errors. At our company, we previously used The message essentially being conveyed here is "Okay there are already X number of ESLint warnings in this package but we won't allow any more", while the message being conveyed with the baseline feature seems to be a more granular version of this: "Okay there are already X number of ESLint errors/warnings for rule R in file F but we won't allow any more." Anyway, it's just a similarity I see, it's not a big deal. |
Probably the main decision to make for this RFC which style of error matching you use. These would be:
With our current implementation, we use the Exact Error Line Matching, which does have some short falls. It works pretty well, the vast majority of PRs don't require any rework. There are cases where if you modify a line at the top of a file though, will cause all of the existing errors to be exposed. This approach was chosen due to simplicity. For the most part this does push a project to having a smaller and smaller baseline, however it does lead to developers touching the baseline "too much" due to them not wanting to fix errors not directly related to their PR. We're currently exploring moving to a context aware error matching, so it only comes up if the nearest 3 lines of code are modified - LuminateOne/eslint-baseline#8. This would provide a solution to the issue above, but it's not yet tested but seems like a sensible approach (if it can be done in a performant way). I would highly recommend an error count approach the same as PHPStan (as per comments from @ondrejmirtes), as that seems to sit with the development team better. Only other question "feature request" I would add into this is to allow for "flushing" a baseline of excess/resolved errors only (and not add any new ones). Just a QOL improvement though as it's typically easy enough to see new lines added into the diff in the baseline. |
@Humni gotcha, that's helpful. So it sounds like the current RFC, which uses error counting, is what you'd prefer. 👍 Keep in mind, too, that we generally build things incrementally. Our goal here is to get a baseline (no pun intended) of support for suppressions such that we can continue to build on top of it as feedback comes in. |
@softius are you still working on this? There is some outstanding feedback to review. |
@kevin-y-ang the baseline doesn't take warnings into consideration or alter warnings in any way. So the message conveyed is ""Okay there are already X number of ESLint errors for rule R in file F but we won't allow any more."" |
Right, and this was my concern -- it feels like
I think this is preferable because it clearly indicates what is happening. |
@fasttime @nzakas I updated the terminology and concept from baseline to suppressions. The new options are now documented for the ESLint CLI. I also separated the execution details from the CLI options for clarity. Implementation notes have been updated too. Having a closer look at the source code, I have noticed that it messages suppressed with |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is looking really good. Just a few minor points to address.
Minor points addressed. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for all your work on this. I'm pretty happy with the current state, so marking as approved.
We still need other TSC members to approve to move forward.
Co-authored-by: Nicholas C. Zakas <[email protected]>
Co-authored-by: Francesco Trotta <[email protected]>
Co-authored-by: Francesco Trotta <[email protected]>
/** | ||
* Checks the provided suppressions against the lint results. | ||
* | ||
* For each error included in `results`, checks if the error and the file are in `suppressions`. | ||
* If yes, the count is decreased by 1 and ignores the error unless the count has reached zero. | ||
* Otherwise, it keeps the error. | ||
* | ||
* It returns the lint results that are not in the suppressions file, | ||
* as well as the unmatched suppressions. | ||
* | ||
* @param {LintResult[]} results The lint results. | ||
* @param {SuppressedViolations} suppressions The suppressions. | ||
* @returns {{ | ||
* results: LintResult[], | ||
* unmatched: SuppressedViolations | ||
* }} | ||
*/ | ||
applySuppressions(results, suppressions) |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the update. Maybe I'm misunderstanding something, so bear with me. It would be helpful if you could add a draft implementation to show how this function will work.
To clarify, this is an example I would like to understand. Suppose you have a file test.js where the no-undef
rule would normally report three errors:
/* eslint no-undef: "error" */
a = b + c;
What will the function return when the suppression file specifies a count
of 2? This is what the arguments will look like:
const results = [{
filePath: '/my-project/test.js',
messages: [{
ruleId: 'no-undef',
severity: 2,
message: "'a' is not defined.",
line: 3,
column: 1,
nodeType: 'Identifier',
messageId: 'undef',
endLine: 3,
endColumn: 2
},
{
ruleId: 'no-undef',
severity: 2,
message: "'b' is not defined.",
line: 3,
column: 5,
nodeType: 'Identifier',
messageId: 'undef',
endLine: 3,
endColumn: 6
},
{
ruleId: 'no-undef',
severity: 2,
message: "'c' is not defined.",
line: 3,
column: 9,
nodeType: 'Identifier',
messageId: 'undef',
endLine: 3,
endColumn: 10
}],
suppressedMessages: [],
errorCount: 3,
fatalErrorCount: 0,
warningCount: 0,
fixableErrorCount: 0,
fixableWarningCount: 0,
source: '/* eslint no-undef: "error" */\n\na = b + c;\n',
usedDeprecatedRules: []
}];
const suppressions = {
"test.js": {
"no-undef": {
count: 2
}
}
};
You can see that the input results
contains details about each error, while the suppressions are just numbers. What will the call to applySuppressions(results, suppressions)
return in this case?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In that case it will return an updated copy of results and an empty object for unmatched (because all suppressions are matched).
// results
const results = [{
filePath: '/my-project/test.js',
messages: [
{
ruleId: 'no-undef',
severity: 2,
message: "'c' is not defined.",
line: 3,
column: 9,
nodeType: 'Identifier',
messageId: 'undef',
endLine: 3,
endColumn: 10
}],
suppressedMessages: [],
errorCount: 1,
fatalErrorCount: 0,
warningCount: 0,
fixableErrorCount: 0,
fixableWarningCount: 0,
source: '/* eslint no-undef: "error" */\n\na = b + c;\n',
usedDeprecatedRules: []
}];
// unmatched
{}
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If I understand correctly, this means removing the first count
messages with a particular rule-id
from each result, where count
is the suppression count. So, in the above example a
and b
would be suppressed but not c
, because in the input result, a
and b
are sorted first in order of location.
I think this behavior would be very surprising to users. This means that some rule violations would be reported only for (the bottom) part of a file.
I think it would be better to report all messages unfiltered if the suppression count is less than the number of messages, because in that situation we can't make a meaningful selection. So for each file and rule-id
we should always report either all messages, or none of them.
@eslint/eslint-tsc looking for another approval on this. |
|
||
### Suppressing violations of a specific rule | ||
|
||
A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this is option can accept an array of string values. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this is option can accept an array of string values. | |
A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this option can accept an array of string values. |
A new option `--suppress-rule [RULE1]` will be introduced to ESLint CLI. When provided, the existing suppressions file will be updated to include any existing violation of the provided rule. The suppressions file will be created if not already exists. Note that this is option can accept an array of string values. | ||
|
||
``` bash | ||
eslint --suppress-rule '@typescript-eslint/no-explicit-any' --suppress-rul '@typescript-eslint/member-ordering' ./src |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
eslint --suppress-rule '@typescript-eslint/no-explicit-any' --suppress-rul '@typescript-eslint/member-ordering' ./src | |
eslint --suppress-rule '@typescript-eslint/no-explicit-any' --suppress-rule '@typescript-eslint/member-ordering' ./src |
* The developer runs `eslint --supress-all ./src` to create the suppressions file. | ||
* Running `eslint ./src` reports no violations and exits with status 0. | ||
* After fixing a violation, the suppressions file still contains the now-resolved violation. | ||
* Running `eslint ./src` again reports no violations but exits with a non-zero status code, indicating the suppressions file needs updating. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In this case, will there be an error message, and what would it look like? Technically, will it be a lint message passed to the formatter along with other lint messages for the file, or a separate output?
That's a good question. I think it makes sense to insert lint messages suppressed by this feature into |
I agree. Makes sense. 👍 |
Makes sense. I think that errors suppressed by the suppressions file will have no |
Summary
Suppress existing violations, so that they are not being reported in subsequent runs. It allows developers to enable one or more lint rules and be notified only when new violations show up.
Related Issues