Add quantile binning for uncertainty estimation including error bound estimation and evaluation metrics #281
Conversation
… index, error bounds, boxplots, cum error, example main
Codecov Report
❗ Your organization is not using the GitHub App Integration. As a result you may experience degraded service beginning May 15th. Please install the Github App Integration for your organization. Read more. @@ Coverage Diff @@
## main #281 +/- ##
==========================================
- Coverage 92.08% 90.86% -1.22%
==========================================
Files 55 63 +8
Lines 5635 6766 +1131
==========================================
+ Hits 5189 6148 +959
- Misses 446 618 +172
|
There was a problem hiding this comment.
You requested my review but the PR is marked as a draft and work in progress so that is a bit confusing. Please refer to other examples and the video https://www.youtube.com/watch?v=I3vifU2rcc0 for standards adopted. Also fix the error reported and missing tests. Then when ready, mark the PR ready for review and request a review from me. Thanks.
|
@Schobs Did you request my review because the template says so? If yes, I should update the template with ifs. |
|
@haipinglu Correct. I followed the template. |
Thanks. I will update the PR template with clearer guidance. |
… index, error bounds, boxplots, cum error, example main
… into landmark_uncertainty
|
@Schobs After checking the details (timeout error), I just rerun your latest PR checks and it passed. Congrats on your first green check in this PR. |
| num_bins: int, | ||
| show_sample_info: str = "None", | ||
| save_path: Optional[str] = None, | ||
| y_lim: int = 120, |
There was a problem hiding this comment.
hard code? Check and log if no time.
There was a problem hiding this comment.
i have added to discussion
| r"$\bf{PSB}$" + ": \n" + r"${} \pm$".format(perc_info[0]) + "\n" + r"${}$".format(perc_info[1]), | ||
| verticalalignment="bottom", # Centered bottom with line | ||
| horizontalalignment="center", # Centered with horizontal line | ||
| fontsize=25, |
There was a problem hiding this comment.
Hard coded. Check what other libraries do with this. If not time to fix it, log it in the discussion.
|
ave might be a good compromise
…On Tue, 18 Jul 2023, 19:31 Lawrence Schobs, ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In kale/evaluate/uncertainty_metrics.py
<#281 (comment)>:
> + weighted_mean_bin = 0.0
+ total_weights_bin = 0.0
+ for l_idx in range(len(bin_accs)):
+ b_acc = bin_accs[l_idx]
+ b_siz = bin_asizes[l_idx]
+ weighted_mean_bin += b_acc * b_siz
+ total_weights_bin += b_siz
+
+ # Avoid div by 0
+ if weighted_mean_bin == 0 or total_weights_bin == 0:
+ weighted_av_bin = 0.0
+ else:
+ weighted_av_bin = weighted_mean_bin / total_weights_bin
+ weighted_av_binwise.append(weighted_av_bin)
+
+ # No weighted average, just normal av
are you advising i change variable names that have av to average e.g.
normal_av_bin_wise -> normal_average_bin_wise ?
—
Reply to this email directly, view it on GitHub
<#281 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFTAQGKR2TZP7ZY4K7SJZZTXQ3JBZANCNFSM5KJDPD3Q>
.
You are receiving this because your review was requested.Message ID:
***@***.***>
|
|
What does "infos" mean?
…On Tue, 18 Jul 2023, 19:38 Lawrence Schobs, ***@***.***> wrote:
***@***.**** commented on this pull request.
------------------------------
In kale/interpret/uncertainty_quantiles.py
<#281 (comment)>:
> + if show_sample_info != "None":
+ flattened_model_data = [x for xss in model_data for x in xss]
+ percent_size = np.round(len(all_b_data) / len(flattened_model_data) * 100, 1)
+ average_samples_per_bin.append(percent_size)
+
+ if show_sample_info == "All":
+ """This adds the number of samples on top of the top whisker"""
+ (x_l, y), (x_r, _) = rect["caps"][-1].get_xydata()
+ x_line_center = (x_l + x_r) / 2
+ all_sample_label_x_locs.append(x_line_center)
+ all_sample_percs.append(percent_size)
+ all_rects.append(rect)
+
+ inner_min_x_loc += 0.1 + width
+
+ """ Keep track of average sample infos. Plot at the END so we know what the max height for all Qs are."""
Please could you clarify what you mean?
—
Reply to this email directly, view it on GitHub
<#281 (comment)>, or
unsubscribe
<https://github.com/notifications/unsubscribe-auth/AFTAQGJWDUE3AU4NGM4OJ4DXQ3J2BANCNFSM5KJDPD3Q>
.
You are receiving this because your review was requested.Message ID:
***@***.***>
|
|
@shuo-zhou Please help resolve the conflict below so that this branch can be updated, which should be straightforward. Thanks. |
|
@Schobs Put the missing test / test problem in discussion list. |
There was a problem hiding this comment.
Most imminent issues have been resolved for merging, with the remaining issues captured in #392 to follow up @alanbijuthomas .
Thank you for contributing such an important tool to PyKale.
haipinglu
changed the title
Implementation of uncertainty estimation for landmark localisation.
Description
Integrates Uncertainty estimation methods including: quantile binning, error bound estimation and evaluation metrics. Provides an example file.
Status
Work in progress
Types of changes
docsmanually updated for new API.