fix: check the number of collected data in post_fp_check_fail

[HuangJiameng]

see #737
dpdispatcher use flag_if_job_task_fail to mark the failed jobs, so post_fp_check_fail can be used before checking the frames. If we use the alternative way the issue mentioned, we should consider different representative outputs according to fp_style. I am wondering if it is a repeat of the following frame checks. However, since flag_if_job_task_fail will be marked as True if one task in the group is failed, I am afraid that rfail could be high when only a few tasks fail. I'd like to ask for some suggestions.

[codecov-commenter]

Codecov Report

Merging #882 (f2df1af) into devel (d55d400) will increase coverage by 0.09%.
The diff coverage is 91.66%.

@@            Coverage Diff             @@
##            devel     #882      +/-   ##
==========================================
+ Coverage   38.12%   38.22%   +0.09%     
==========================================
  Files          99       99              
  Lines       17782    17823      +41     
==========================================
+ Hits         6779     6812      +33     
- Misses      11003    11011       +8     

Impacted Files	Coverage Δ
dpgen/generator/run.py	62.24% <91.66%> (+0.02%)	⬆️
dpgen/dispatcher/AWS.py	25.26% <0.00%> (-2.87%)	⬇️
dpgen/auto_test/Elastic.py	62.69% <0.00%> (-0.33%)	⬇️
dpgen/tools/relabel.py	14.60% <0.00%> (-0.14%)	⬇️
dpgen/generator/arginfo.py	100.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

[njzjz]

This is not correct as dpdispatcher does not allow a command to fail. It will retry and raise RuntimeError. If you want to allow some command to return a non-zero exit code, you can consider executing some_command || touch some_flag and then count some_flag instead.

[HuangJiameng]

How about counting *_task_tag_finished? Each task has task_tag_finished. $rfail = 1 - \frac{nfinished}{ntasks}$
Creating a new tag seems cumbersome.

[njzjz]

Dpdispatcher expects all tasks finished.

[HuangJiameng]

Then post_fp_check_fail seems useless? Is it reasonable to discard it? It doesn't work now in fact.

[njzjz]

It is used with the old dispatcher.

[njzjz]

Also, I think the tag will not be backward unless it is added into the list of backward files.

[HuangJiameng]

Since dpdispatcher will raise an error if any task fails, while the ratio of fail frames shows available data, shall we deprecate post_fp_check_fail?

[njzjz]

One can use some_command || : in the command to allow some_command to fail (but the whole command still successes). And dpdata will skip non-coverage data. After new data is collected, we can count the number of data. The failed ratio can be calculated as what I mentioned in #737.

[HuangJiameng]

check fail according to the number of collected data

[HuangJiameng]

Weirdly, unit tests are passed locally. @njzjz Do you have any idea?

[njzjz]

It looks like the failed unit test outputs a MultiSystems directory. Your method does not cover this situation. You may use dpgen.util.expand_sys_str.

[njzjz]

LGTM