[Feature] a toMatchScreenshot() for comparing two different screenshots

[NVolcz]

We currently have expect(screenshot).toMatchSnapshot(name[, options]) which is nice when comparing against at snapshot but it doesn't work when comparing a screenshot against a reference website which is useful for doing visual regression to see what has changed. I propose that we add a toMatchScreenshot(screenshot[, options]) that could be used for something like this:

page.goto("production.example.com")
const ref = await page.screenshot({
  animations: "disabled",
  fullPage: true,
  scale: "css"
});

page.goto("localhost:8080")
const test = await page.screenshot({
  animations: "disabled",
  fullPage: true,
  scale: "css"
});

expect(test, "should  match reference").toMatchScreenshot(ref, {
  maxDiffPixelRatio: 0,
  maxDiffPixels: 0,
  threshold: 0
});

[dgozman]

@NVolcz I think we can probably do something like expect(image1).toMatchImage(image2) for this usecase. See also #18897.

In your particular case, I can recommend to produce screenshots once in production environment, commit them to your version control, and then compare localhost screenshots to these expectations. This way, you'll have a stable set of screenshots to test against. I hope this helps.

[NVolcz]

expect(image1).toMatchImage(image2) would be awesome!
I implemented something like what you are suggesting but handling snapshots is troublesome and doesn't seem worth it in my case.

[thenbe]

@dgozman

Would you consider implementing this in a way that would support the following usecase?

test('MyButton has no side effects', async ({ page }) => {
	await page.goto('/');

	const before = await page.getByTestId('my-element').screenshot();

	await page.getByRole('button', { name: 'MyButton' }).click();

	const after = await page.getByTestId('my-element').screenshot();

        // ensure my-element still looks the same
	await expect(before).toMatchImage(after);
});

Specifically:

• Compare a screenshot taken at the beginning of the test with one taken at the end.
• Without saving them or having to handle paths, etc.
• While benefiting from the 'visual image diff widget' in the test report which makes it easy to find out why a particular test failed. Like the videos, the images can be retained-on-failure, and discarded on success.

[dgozman]

@ambiguous48 Yes, so far I think that toMatchImage() could support both buffers and file paths.

[NikkTod]

Hello @aslushnikov, is there a chance we see that feature in 1.29?

[aslushnikov]

@NikkTod no, it's not coming to 1.29. Will be moved to 1.30 instead

[elesh90]

Does this feature work in version 1.32 by now?

[AleksandrovAlex]

It will be very helpful to have this feature implemented because it opens a wide range of new options. Plus the new masking trick ... Can't wait.

[danilovxp]

Hello @aslushnikov,

Is there any lucky news about this feature? 🙏