[Feature] Playwright integration with Azure DevOps Test Plans #14839

PavanMudigondaTR · 2022-06-13T17:50:28Z

Is it possible to provide support for updating test results in Azure DevOps Test Plans by using Playwright directly ? I have come across this feature that is provided in Webdriver.IO. The test case execution status for Automated test cases are currently having to be updated manually as either passed or failed. And also the Automation status drop down we are not able to update it as Automated.

https://webdriver.io/docs/gmangiapelo-wdio-azure-devops-service
https://github.com/gianlucamangiapelo/wdio-azure-devops-service/tree/main/.github/workflows

dgozman · 2022-06-14T20:55:15Z

Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.

PavanMudigondaTR · 2022-06-15T10:01:59Z

Thanks a lot for quick reply! I will give this a try!

PavanMudigondaTR · 2022-06-15T15:03:31Z

Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.

@dgozman It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?

dgozman · 2022-06-15T16:32:14Z

It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?

Well, this is actually an integration between the test runner and ADO. @playwright/test for JS/TS includes a test runner, and this reporter integrates with ADO.

For python we recommend pytest, so you will need pytest+ADO integration, perhaps something like this?

Similarly, for Java you would need something that brings together JUnit+ADO, and for C# something that supports MSTest or NUnit with ADO.

PavanMudigondaTR · 2022-06-15T18:55:23Z

It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?

Well, this is actually an integration between the test runner and ADO. @playwright/test for JS/TS includes a test runner, and this reporter integrates with ADO.

For python we recommend pytest, so you will need pytest+ADO integration, perhaps something like this?

Similarly, for Java you would need something that brings together JUnit+ADO, and for C# something that supports MSTest or NUnit with ADO.

@dgozman I get it now. Playwright Test is only Js/TS Runner. I understand. So we need similar solutions for Pytest, NUnit and Junit5 as well now. I would appreciate if this can be looked at playwright team or some community members who can ? I see Azure DevOps has node api, python api, C# api and go api which can be used to build this integration. But i am not that skilled to do this. The pip library link you sent in previous reply will not solve it fully.

daniavander · 2022-08-24T14:06:25Z

how is it fill:

daniavander · 2022-08-25T06:53:16Z

I get this in the log
azure: Test [31035] Smoke test - Activity list @response - passed
azure: Start publishing: [31035] Smoke test - Activity list @response
SUITE PASS :)
azure: Start upload attachments for test case [31035]
azure: Result published: [31035] Smoke test - Activity list @response
azure: Run 1000284 - Completed

but i see nothing in test case

PavanMudigondaTR · 2022-08-25T11:33:18Z

I get this in the log

azure: Test [31035] Smoke test - Activity list @response - passed

azure: Start publishing: [31035] Smoke test - Activity list @response

SUITE PASS :)

azure: Start upload attachments for test case [31035]

azure: Result published: [31035] Smoke test - Activity list @response

azure: Run 1000284 - Completed

but i see nothing in test case

Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!

daniavander · 2022-08-25T12:18:28Z

o i see, so we are waiting it !

mechandan · 2022-11-08T12:14:08Z

@dgozman Any official update on integration playwright typescript/javascript with the ADO test plan ?? Here is a third-party reporter it just take a single plan id ( suite id ) at a time , in every sprunt the suite id changes , ( much appreciate the effort from the developer n thankyou! )

daniavander · 2023-01-20T12:18:56Z

Is there any progression with playwright automated test results connect with azure devops manual test cases?

joeflan · 2023-02-02T20:02:10Z

Yeah I'd also

Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!

Is there any place to track this, or any idea on when this integration might happen?

rowel14 · 2023-02-16T14:58:21Z

I get this in the log
azure: Test [31035] Smoke test - Activity list @response - passed
azure: Start publishing: [31035] Smoke test - Activity list @response
SUITE PASS :)
azure: Start upload attachments for test case [31035]
azure: Result published: [31035] Smoke test - Activity list @response
azure: Run 1000284 - Completed
but i see nothing in test case

Can you check Test Runs ? Also please note this Azure DevOps TestPlans is being integrated in to official release so we might see it soon!

Do we have update on this one? I got the same result. My test case didn't update

rowel14 · 2023-02-17T11:30:12Z

Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.

@dgozman It worked fine in my Playwright TypeScript. I was wondering if it can be used similarly in Python, Java, C# ? Cause I need to provide solutions to my whole organization. Does this require official Playwright support to use in other language bindings ?

could you share some screenshots on how you wer able to work it?

PavanMudigondaTR · 2023-02-17T14:05:27Z

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.

Please note if you are committing your repo to GitHub make sure use secrets

Instead of passing token directly use secrets.

INSTEAD OF DOING:

token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',

DO:

token: process.env.ADO_TOKEN

to run it locally

export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds

FINALLY in GitHub Actions you can do like this

pass in

name: run playwright tests
env:
ADO_TOKEN: ${{ secrets.ADO_TOKEN }}
shell: bash
run: npx playwright test

rowel14 · 2023-02-17T14:29:43Z

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.

Please note if you are committing your repo to GitHub make sure use secrets

Instead of passing token directly use secrets.

INSTEAD OF DOING:

token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',

DO:

token: process.env.ADO_TOKEN

to run it locally

export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds

FINALLY in GitHub Actions you can do like this

pass in

name: run playwright tests
env:
ADO_TOKEN: ${{ secrets.ADO_TOKEN }}
shell: bash
run: npx playwright test

I'm actually using Azure DevOps, I'm still struggling to update the result of a specific test case in my test plan in Azure Test Plan,

rowel14 · 2023-02-17T14:31:39Z

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.

Please note if you are committing your repo to GitHub make sure use secrets

Instead of passing token directly use secrets.

INSTEAD OF DOING:

token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',

DO:

token: process.env.ADO_TOKEN

to run it locally

export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds

FINALLY in GitHub Actions you can do like this

pass in

name: run playwright tests
env:
ADO_TOKEN: ${{ secrets.ADO_TOKEN }}
shell: bash
run: npx playwright test

I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.

PavanMudigondaTR · 2023-02-17T15:29:51Z

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.
Please note if you are committing your repo to GitHub make sure use secrets
Instead of passing token directly use secrets.
INSTEAD OF DOING:
token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',
DO:
token: process.env.ADO_TOKEN
to run it locally
export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds
FINALLY in GitHub Actions you can do like this
pass in

name: run playwright tests
env:
ADO_TOKEN: ${{ secrets.ADO_TOKEN }}
shell: bash
run: npx playwright test

I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.

Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman

I am using a tool called SpecSync currently due to few problems I faced.

rowel14 · 2023-02-17T15:59:53Z

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.
Please note if you are committing your repo to GitHub make sure use secrets
Instead of passing token directly use secrets.
INSTEAD OF DOING:
token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',
DO:
token: process.env.ADO_TOKEN
to run it locally
export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds
FINALLY in GitHub Actions you can do like this
pass in

name: run playwright tests
env:
ADO_TOKEN: ${{ secrets.ADO_TOKEN }}
shell: bash
run: npx playwright test

I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.

Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman

I am using a tool called SpecSync currently due to few problems I faced.

Yeah, this is what I am talking about, I want to update the outcome (result) of each test cases in my test plan in Azure Test Plan. Do we have already a solution for this? Does Playwright will support this thing soon? Any alternative that we can use?

Guru426 · 2023-04-18T19:36:05Z

@Rowel @PavanMudigondaTR

Alex has pretty good documentation which I simply followed and was able to make it work in like 10 minutes only.
Please note if you are committing your repo to GitHub make sure use secrets
Instead of passing token directly use secrets.
INSTEAD OF DOING:
token: ‘asasdskjdklfjdkljfdfjdflkdjflddsfdfds',
DO:
token: process.env.ADO_TOKEN
to run it locally
export ADO_TOKEN=asasdskjdklfjdkljfdfjdflkdjflddsfdfds
FINALLY in GitHub Actions you can do like this
pass in

name: run playwright tests
env:
ADO_TOKEN: ${{ secrets.ADO_TOKEN }}
shell: bash
run: npx playwright test

I was able to publish the result of test runs but updating the result of a specific test case which I link to my playwright automation test using [testCaseId] is not working.

Are looking for way to reflect the status in TestPlans ? I I am not able to do this either. @alexneo2003 @dgozman
I am using a tool called SpecSync currently due to few problems I faced.

Yeah, this is what I am talking about, I want to update the outcome (result) of each test cases in my test plan in Azure Test Plan. Do we have already a solution for this? Does Playwright will support this thing soon? Any alternative that we can use?

Hello guys, Were you able to update the results to the test plans in Azure now ? If so, kindly share few inputs on how did you achieved it when you get sometime. Thanks!

ravindrakumar6 · 2023-04-20T05:07:50Z

getting this error while results got published,

this is my test case automation status in TFS

ravindrakumar6 · 2023-04-20T05:08:13Z

Can you please help me on this

daniavander · 2023-04-20T10:39:23Z

here is my log:
azure: Uploading attachments for test: [31036] - Smoke test - Reports page @response @smok
azure: Uploaded attachment
azure: Left to publish:
azure: Test results published for 14 test(s
azure: Run 1000348 - Complete

where set the automation status and automation complexity?
are these own fields no?

ravindrakumar6 · 2023-04-21T05:58:10Z

@daniavander ,

Please find these screenshots hope this helps

TFS Automation Options available

MyPlaywright Config

Test Cases -- see the error down below

Same Test Case ID i used

See these details too

After analyzing all these still something is missing or not sure where exactly it's going wrong
if you get any idea great help.

ravindrakumar6 · 2023-04-21T06:01:54Z

@daniavander meanwhile please check this one provided by @alexneo2003

https://docs.specsolutions.eu/specsync/contact/troubleshooting#test-result-publishing-fails-with-mismatch-in-automation-status-of-test-case-and-test-run

so still trying to figure it out why this error is appearing. Please try to help if any ideas or hint that can resolve this issue Thanks.

daniavander · 2023-04-21T10:17:03Z

@ravindrakumar6 sorry, i do not know what is the problem.
In ma devops i have only two status for automation status

in the config i do not add the suite id just the planid, perhaps

daniavander · 2023-04-21T11:26:36Z

@aslushnikov are there any future time, when would you like to take this integration to the official release?

thomas-phillips-nz · 2023-05-26T01:47:18Z

I tried asking questions of the playwright team in every livestream I could during Microsoft Build but it sounds like the Playwright team aren't familiar with Azure Test Plans. They kept referring to running tests in Azure Devops CI/Pipelines, but it doesn't sound like it will work with Azure Test Plans.

Would be good if you as a Microsoft team could support the Azure Test Plans feature.

PavanMudigondaTR · 2023-05-26T02:28:55Z

I tried asking questions of the playwright team in every livestream I could during Microsoft Build but it sounds like the Playwright team aren't familiar with Azure Test Plans. They kept referring to running tests in Azure Devops CI/Pipelines, but it doesn't sound like it will work with Azure Test Plans.

Would be good if you as a Microsoft team could support the Azure Test Plans feature.

There are two options currently. I recommend you try out in below sequence.

Oscaruzzo · 2023-07-06T08:40:40Z

This would be incredibly useful. I'm using the third-party reporter other people mentioned, at the moment https://github.com/alexneo2003/playwright-azure-reporter but a "built-in" integration would be awesome. Also integrated in the VSCode pluging, hopefully.

nguyenquangtay · 2023-09-26T02:32:09Z

upvote, thanks!

HannahTran96 · 2023-10-11T15:16:42Z

Here is a third-party reporter that integrates with ADO. While we did not finalize our plans on integration with ADO, you could give that reporter a try.

Hi, could you support me, I always got No test points found for test case [4526] associated with test plan 4496. Check, maybe testPlanId, what you specified, is incorrect although my testPlanId and my testcase id is correct. Could you help me point out what part I should fix
@dgozman []

alexneo2003 · 2023-10-11T15:41:20Z

Hi, could you support me, I always got No test points found for test case [4526] associated with test plan 4496. Check, maybe testPlanId, what you specified, is incorrect although my testPlanId and my testcase id is correct. Could you help me point out what part I should fix @dgozman []

Read this , pls

alexneo2003 · 2023-10-11T15:41:28Z

Hi, could you support me, I always got No test points found for test case [4526] associated with test plan 4496. Check, maybe testPlanId, what you specified, is incorrect although my testPlanId and my testcase id is correct. Could you help me point out what part I should fix @dgozman []

Read this , pls

HannahTran96 · 2023-10-13T04:09:45Z

@

Hi, could you support me, I always got No test points found for test case [4526] associated with test plan 4496. Check, maybe testPlanId, what you specified, is incorrect although my testPlanId and my testcase id is correct. Could you help me point out what part I should fix @dgozman []

Read this , pls

Thank you, it works. I have question I want to combine playwright report and cucumber but I dont know how to do it, I tried some ways but it does not work. Could you help me @alexneo2003

justinteix · 2024-03-29T02:24:51Z

Hello, are there any updates on whether Microsoft is still looking to integrate Azure DevOps Test Plans into Playwright built-in? I'm new to Playwright and really would like to update my test case executions on my ADO test plans automatically during my CI runs. I've used other tools that have had this feature for a while and would really love to see it in Playwright.

WPatryk · 2024-05-23T07:45:06Z

Hello @aslushnikov , do you know if there is any plan to implement this anytime soon? Is there anything we could do to make it possible for near future? It looks like a lot of people would find this useful. It would be a great addition to PW integration with ADO.

jonathanrossTPA · 2024-05-23T11:33:15Z

I am hoping that Playwright recorder is also added to the Azure Devops Test & Feedback browser add-on, which would be a logical solution

…

On Thu, 23 May 2024 at 10:45, WPatryk ***@***.***> wrote: Hello @aslushnikov <https://github.com/aslushnikov> , do you know if there is any plan to implement this anytime soon? Is there anything we could do to make it possible for near future? It looks like a lot of people would find this useful. — Reply to this email directly, view it on GitHub <#14839 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AUJM4YC6THHZMAETGFVC2MDZDWNBVAVCNFSM5YVAFTMKU5DIOJSWCZC7NNSXTN2JONZXKZKDN5WW2ZLOOQ5TEMJSGY2DINJUGQ3A> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

-- This email and any files transmitted with it contain confidential information and/or privileged or personal advice. This email is intended for the addressee(s) stated above only. If you are not the addressee of the email please do not copy or forward it or otherwise use it or any part of it in any form whatsoever. If you have received this email in error please notify the sender and remove the e-mail from your system. Thank you. This is an email from the company Just Eat Takeaway.com N.V., a public limited liability company with corporate seat in Amsterdam, the Netherlands, and address at Piet Heinkade 61, 1019GM Amsterdam, registered with the Dutch Chamber of Commerce with number 08142836 and where the context requires, includes its subsidiaries and associated undertakings.

dgozman added the feature-test-runner Playwright test specific issues label Jun 14, 2022

yury-s mentioned this issue Nov 8, 2022

Support for test plan ( ADO )integrations with playwright javascript/ typescript #18644

Closed

aslushnikov mentioned this issue Jan 11, 2023

[Feature] Adding a dashboard-like integration with Azure DevOps to view test results #20028

Closed

aslushnikov mentioned this issue Jun 7, 2023

[Feature] VS Code extension- Link automated test with an Azure DevOps test case #23527

Closed

yury-s mentioned this issue Aug 2, 2023

[Feature] Support for associating specific test script to test case in Azure DevOps Test Plan #24550

Closed

[Feature] Playwright integration with Azure DevOps Test Plans #14839

[Feature] Playwright integration with Azure DevOps Test Plans #14839

Comments

PavanMudigondaTR commented Jun 13, 2022 • edited Loading

dgozman commented Jun 14, 2022

PavanMudigondaTR commented Jun 15, 2022

PavanMudigondaTR commented Jun 15, 2022

dgozman commented Jun 15, 2022

PavanMudigondaTR commented Jun 15, 2022

daniavander commented Aug 24, 2022

daniavander commented Aug 25, 2022

PavanMudigondaTR commented Aug 25, 2022

daniavander commented Aug 25, 2022

mechandan commented Nov 8, 2022

daniavander commented Jan 20, 2023

joeflan commented Feb 2, 2023

rowel14 commented Feb 16, 2023

rowel14 commented Feb 17, 2023

PavanMudigondaTR commented Feb 17, 2023 • edited Loading

rowel14 commented Feb 17, 2023

rowel14 commented Feb 17, 2023

PavanMudigondaTR commented Feb 17, 2023

rowel14 commented Feb 17, 2023

Guru426 commented Apr 18, 2023 • edited Loading

ravindrakumar6 commented Apr 20, 2023

ravindrakumar6 commented Apr 20, 2023

daniavander commented Apr 20, 2023

ravindrakumar6 commented Apr 21, 2023

ravindrakumar6 commented Apr 21, 2023

daniavander commented Apr 21, 2023

daniavander commented Apr 21, 2023

thomas-phillips-nz commented May 26, 2023

PavanMudigondaTR commented May 26, 2023

Oscaruzzo commented Jul 6, 2023

nguyenquangtay commented Sep 26, 2023 • edited Loading

HannahTran96 commented Oct 11, 2023 • edited Loading

alexneo2003 commented Oct 11, 2023

alexneo2003 commented Oct 11, 2023

HannahTran96 commented Oct 13, 2023 • edited Loading

justinteix commented Mar 29, 2024 • edited Loading

WPatryk commented May 23, 2024 • edited Loading

jonathanrossTPA commented May 23, 2024 via email

PavanMudigondaTR commented Jun 13, 2022 •

edited

Loading

PavanMudigondaTR commented Feb 17, 2023 •

edited

Loading

Guru426 commented Apr 18, 2023 •

edited

Loading

nguyenquangtay commented Sep 26, 2023 •

edited

Loading

HannahTran96 commented Oct 11, 2023 •

edited

Loading

HannahTran96 commented Oct 13, 2023 •

edited

Loading

justinteix commented Mar 29, 2024 •

edited

Loading

WPatryk commented May 23, 2024 •

edited

Loading