Research, Select, & Implement E2E Testing Framework

[tn3rb]

A critically important aspect of our goal of achieving a fully automated CI/CD pipeline from development to final production product, is having automated End to End (E2E) testing in place to protect against unintended changes and breakage.

steps

• research available E2E frameworks and document their features, functionality, costs, etc, essentially understand their various pros and cons. Discussion can be done within this issue.

• select framework based on above discussion

• execute changes via multiple PRs in a granular fashion

• implement E2E testing which will involve the following:

  • cutting FET/e2e-tests branch from master

  • creating any service accounts required by e2e framework

  • installing dependencies for e2e framework

  • initial configuration, etc

  • test creation (ongoing process)

• integrate automated testing and automatic deployment into CI/CD pipeline, which involves:

  • creating a new branch named something like production or deploy cut from master

  • editing of GitHub Actions (GHA) pertaining to Chromatic (StoryBook) so that they only run upon pushes to the production / deploy branch

  • creation of new GHA for running the new E2E tests upon pushes to the production / deploy branch

  • creation of new GHA for automatically rebasing production / deploy branch on to master when ALL tests pass

So the new development process will look something like this:

[nicolad]

GB-related context:

• they're using puppeteer for e2e: https://github.com/WordPress/gutenberg/tree/master/packages/e2e-tests
• they have some nice utils related to this ☝️ package that we can reuse, if we go with puppeteer: https://github.com/WordPress/gutenberg/tree/master/packages/e2e-test-utils/src

[tn3rb]

how coupled are those utilities to the WP core redux stores and would we still be able to utilize them if they are heavily coupled?

[nicolad]

how coupled are those utilities to the WP core redux stores and would we still be able to utilize them if they are heavily coupled?

yes, in e2e-test-utils there is a bit of usage of core dispatchers:

[nicolad]

I wanted to follow the GB's decision path and it led me to some interesting findings:

• Testing: Experiment with Puppeteer for E2E testing WordPress/gutenberg#5618 (comment)
• Try running our e2e tests using playwright WordPress/gutenberg#23238

[nicolad]

these are the main competitors:

[nicolad]

interesting related info, today Garth had reported an issue specific to safari only which is happening on his Mac,
I've noticed that it's possible to run e2e tests on macos-latest in playwright GH action

[nicolad]

Auto-waiting feature is one of the main differences between playwright and puppeteer: https://playwright.dev/docs/actionability/
which makes these type of checks not necessary: https://github.com/WordPress/gutenberg/blob/master/packages/e2e-test-utils/src/install-plugin.js#L23

[nicolad]

pupetteer and playwright execution time is almost the same:

from https://blog.checklyhq.com/cypress-vs-selenium-vs-playwright-vs-puppeteer-speed-comparison/#conclusion

[nicolad]

another interesting aspect I found about playwright is that it has support for android emulation and other flavors:

---

this will be important when we will start using react native