MM-23788 Setup for visual regression testing #5783
Conversation
There was a problem hiding this comment.
LGTM. All tests passed. Just a quick question, though, about applitools config file (see comment in e2e/applitools.config.js).
I used this locally to run several times,
APPLITOOLS_API_KEY=XXXXXXXX APPLITOOLS_BATCH_NAME="Webapp - master" ENABLE_VISUAL_TEST=true node run_tests.js --group='@visual_regression'
Also, I'm seeing this warning when I run the test locally. I haven't looked much into it but just curious if this is a security issue when running with a specific node version (I use v10.19.0)
(node:87342) Warning: Setting the NODE_TLS_REJECT_UNAUTHORIZED environment variable to '0' makes TLS connections and HTTPS requests insecure by disabling certificate verification.
| module.exports = { | ||
| appName: 'Mattermost Webapp UI', | ||
| accessibilityValidation: {level: 'AA', guidelinesVersion: 'WCAG_2_0'}, | ||
| batchName: 'Webapp - master (dev)', |
There was a problem hiding this comment.
Quick question: is batchName here supposed to be a default value? Reason I asked is when I don't set APPLITOOLS_BATCH_NAME, the value of the batch name in the UI reports is undefined.
There was a problem hiding this comment.
Oh, good catch. Yes, that should be the default value. Should be fixed now, PTAL.
|
@josephbaylon re: NODE_TLS_REJECT_UNAUTHORIZED, I'm not particularly sure what's causing it and the implication of it. I'm seeing that too locally but not in CircleCI, even if both are on the same version of node |
There was a problem hiding this comment.
This is really cool! 🎉
In case my experience running this for the first time helps:
- For some reason I had not executed
npm i, so on the first run I was getting really weird errors. Maybe it's interesting to put it in the docs. - Then the tests failed because Chrome was not installed, although Cypress has always worked with Chromium for me. I installed Chrome and everything worked smoothly.
- After the test run, I checked Applitools and I saw that I only had the snapshots of three batch runs, which was a bit confusing because I had run 4 tests. Then I realized that the
System Console - Enterprisetest has no snapshots whatsoever. I left a comment there.
I'm not an expert here, but the code looks good, so I'm approving. Thank you!
| before(() => { | ||
| // * Check if server has license for feature | ||
| cy.requireLicenseForFeature('Elasticsearch'); | ||
|
|
||
| // # Go to system admin then verify admin console URL and header | ||
| cy.visit('/admin_console/about/license'); | ||
| cy.url().should('include', '/admin_console/about/license'); | ||
| cy.get('.admin-console', {timeout: TIMEOUTS.HALF_MIN}).should('be.visible').within(() => { | ||
| cy.get('.admin-console__header').should('be.visible').and('have.text', 'Edition and License'); | ||
| }); | ||
| }); | ||
|
|
||
| testCases.forEach((testCase) => { | ||
| it(`${testCase.section} - ${testCase.header}`, () => { | ||
| // # Click the link on the sidebar | ||
| cy.get('.admin-sidebar').should('be.visible').within(() => { | ||
| cy.findByText(testCase.sidebar).scrollIntoView().should('be.visible').click(); | ||
| }); | ||
|
|
||
| // * Verify that it redirects to the URL and matches with the header | ||
| cy.url().should('include', testCase.url); | ||
| cy.get('.admin-console').should('be.visible').within(() => { | ||
| cy.get('.admin-console__header').should('be.visible').and(testCase.headerContains ? 'contain' : 'have.text', testCase.header); | ||
| }); | ||
| }); | ||
| }); | ||
| }); |
There was a problem hiding this comment.
There is no visual snapshot in this file. Should it be outside of the @visual_regression group?
There was a problem hiding this comment.
Good catch! I also realized that this was unintentionally checked in since I have it on my on going dev but it's fine to remain as is. I will have follow up PRs and will add corresponding metadata and visual tests.
|
@agarciamontoro Thanks for running the tests and valuable inputs. |
Probably not, when I first run an e2e test I followed that guide and everything was clear. I added that comment in case we have different docs for the visual regression testing. But if that's the entry point, it's perfect the way it is now. Thanks! |
Summary
Integrate visual regression testing using Applitools with Cypress.
How to run visual regression testing?
The feature is disabled by default.
To be able to run, you may get API key from Applitools and set into your env variable, and then run:
ENABLE_VISUAL_TEST=true node run_tests.js --group='@visual_regression'Note:
/visual_regression) from functional tests:How to get screenshot from the spec file?
There are three steps to get the screenshot.
cy.visualEyesOpen()cy.visualSaveSnapshot()cy.visualEyesClose()Ticket Link
JIRA ticket: https://mattermost.atlassian.net/browse/MM-23788