about porting genutil.statistics.linearregression function

[bonfils2]

Hi all, I think that it would a great idea to convert (or plan to convert) the “genutil.statistics.linearregression” function from cdat (https://cdat.llnl.gov/documentation/utilities/utilities-2.html ) into xcdat. This function was very useful, easy to use on a 3D field, and loaded with options to compute the p-value with or without adjustment for standard error or critical t-value (a calculation that was implemented years ago in cdat after one of Ben Santer’s papers on that topic).

[pochedls]

@bonfils2 - thanks for raising this. One quick question – the slope/intercept is pretty straightforward to calculate with existing libraries – it seems like the main addition would be the error metrics (if existing libraries don't do this).

If I had x[time] and y[time, lat, lon], this function would return a slope[lat, lon], intercept[lat, lon], and p-value field [lat, lon]. Is that right?

@tomvothecoder - can this discussion stay here or should it move to one of the feature discussion threads?

[tomvothecoder]

Thanks for this @bonfils2. @pochedls I think it's fine to keep this discussion here since it'll probably dive deeper into the technical details.

I linked this post to that thread #271 (reply in thread).

[bonfils2]

@pochedls

If I had x[time] and y[time, lat, lon], this function would return a slope[lat, lon], intercept[lat, lon], and p-value field [lat, lon]. Is that right?
Yes, that is right! For the heads-up, Sam is now trying to port this cdat function into xcdat.

[Xunius]

Just to get some updates: are we getting some progress in porting the genutil tools to xcdat? The genutil.statistics module provides some very helpful tools for computing statistics, e.g. std, variances, etc., with proper areal/temporal weighting. I think it would serve as good complements to the existing spatial/temporal averaging utilities in xcdat.

[pochedls]

Thanks for your comment @Xunius. There are not plans currently to port genutil.statistics to xcdat, though I think @bonfils2 may have worked with others to create a linearregression function that replicates key CDAT features into xcdat.

I think one unanswered question that I have is: do the statistical functions in genutil.statistics already exist (e.g., in scipy or sklearn or other packages)? It would be very helpful if you could comment on geo-centric statistical tools that are missing (and what functions are particularly useful) in the New Features discussion.

[lee1043]

@Xunius @pochedls thank you for the discussion. In PMP, I added functions to calculate statistics including the followings:

• bias
• pattern correlation
• mean absolute error
• root mean square error (RMSE)
• Centered RMSE
• standard deviation

I'd be happy to bringing these capabilities to xcdat if we decide that would be useful.

Alternatively, we can advise users to check with the package called xskillscore, which can calculate below statistics.

• Correlation-Based

  • Effective Sample Size (effective_sample_size)
  • Pearson Correlation (pearson_r)
  • Pearson Correlation effective p value (pearson_r_eff_p_value)
  • Pearson Correlation p value (pearson_r_p_value)
  • Slope of Linear Fit (linslope)
  • Spearman Correlation (spearman_r)
  • Spearman Correlation effective p value (spearman_r_eff_p_value)
  • Spearman Correlation p value (spearman_r_p_value)

• Distance-Based

  • Coefficient of Determination (r2)
  • Mean Absolute Error (mae)
  • Mean Absolute Percentage Error (mape)
  • Mean Error (me)
  • Mean Squared Error (mse)
  • Median Absolute Error (median_absolute_error)
  • Root Mean Squared Error (rmse)
  • Symmetric Mean Absolute Percentage Error (smape)

[tomvothecoder]

At our Dev Day Meeting, let's make a final determination on if statistical functionality is in the scope of xCDAT or if there are other packages available out there to use.