Attach failing plots to CI results

[ivirshup]

If plots fail on CI, this should let us see the expected, actual, and diff through azure. Hopefully It works!

I think this could make debugging plotting issues much easier. Here's an example of what the results look like:

Current issue, if tests are run in parallel, this does not work (pytest-dev/pytest-nunit#40), which is not an immediate problem for CI, but limits applicability for local usage. I believe this is an issue with this particular pytest plugin, not necessarily this strategy.

[ivirshup]

yes, they are annoying particularly because is not possible to actually check why a test failed on the server while passes locally.

@fidelram, from your comment (#1551 (comment)), makes me think you'd like to enable this? If you okay this, all this needs to be ready to merge is:

• Figure out where result xml should live
• .gitignore update
• Remove failing test (just there as an example)
• Document where to find this stuff

[fidelram]

This is great Isaac! Very welcome feature. Took me a while to find the 'attachments' in the azure webpage.

Regarding the location or results.xml, I honestly don't have an opinion were to locate it.

[ivirshup]

Great!

Took me a while to find the 'attachments' in the azure webpage

I'll document this!

[ivirshup]

@meeseeksdev backport to 1.7.x

[lumberbot-app]

Owee, I'm MrMeeseeks, Look at me.

There seem to be a conflict, please backport manually. Here are approximate instructions:

1. Checkout backport branch and update it.

$ git checkout 1.7.x
$ git pull

2. Cherry pick the first parent branch of the this PR on top of the older branch:

$ git cherry-pick -m1 5fc12f4a918e21f0c57937b787d52040db046f01

3. You will likely have some merge/cherry-pick conflict here, fix them and commit:

$ git commit -am 'Backport PR #1587: Attach failing plots to CI results'

4. Push to a named branch :

git push YOURFORK 1.7.x:auto-backport-of-pr-1587-on-1.7.x

5. Create a PR against branch 1.7.x, I would have named this PR:

"Backport PR #1587 on branch 1.7.x"

And apply the correct labels and milestones.

Congratulation you did some good work ! Hopefully your backport PR will be tested by the continuous integration and merged soon!

If these instruction are inaccurate, feel free to suggest an improvement.

[gokceneraslan]

@ivirshup do you know why are there sometimes three:

and sometimes two plots: