More overview documentation would be very helpful

[rjplevin]

I may have missed it, but I haven't found documentation of the semantics of each method and its various outputs. I was hoping to not have to read all the original papers just to use the library.

The example at http:https://salib.readthedocs.io/en/latest/basics.html gives this kind of information for the Sobol' method. A similar discussion for the other methods, and a comparison of methods (which to use, when?) would also be helpful.

Thanks!

[willu47]

Thanks for the feedback @rjplevin. I think this could go one of two ways. The methods themselves tend to be quite involved, despite the uniform interface that SALib provides to each of them, and understanding the sensitivity indices obtained from the methods requires some level of understanding of the methods - for which the papers may be the best source.

However, I'm definitely supportive of SALib becoming a not only easy to use, but also a source of new understanding about the methods and their similarities and differences. The question is how to go about this. As noted above, the interface to each of the methods is very similar - you could easily swap in Morris instead of Sobol in the example to which you provided a link. And a cursory description of the sensitivity indices is included in the docstrings of each of the methods. Could these be expanded to provide more information about interpretation?

[rjplevin]

Hi, Will. I agree that the papers are ultimately the best source, but this is puts a lot of work on the user to use a library. I've been using a Monte Carlo approach of my own design in Python with a rank-correlation-based SA method a la Crystal Ball. I'm integrating SALib into my system to allow use of its sampling and analysis methods within my framework. Several things aren't clear to, and I was hoping not to have to read all the papers to figure this out. - Which methods are most appropriate for which SA objectives? - What are the main benefits of each method / differences among them? - How big an N is required for each method to produce meaningful results, and how does this depend on the number of random variables? (I'm running a model that takes 10 min to solve on a Linux cluster, so a few thousand runs are manageable, but 40,000 becomes a challenge.) - What would be the best way (i.e., which SALib method, or perhaps my Monte Carlo method) would be best to reduce the number of important variables and thus to reduce N for the more detailed analysis? - How does one interpret the results? I imagine this could be written up in a few paragraphs for each method. Thanks again, Rich =============================== Richard Plevin, PhD Transportation Sustainability Research Center Institute of Transportation Studies University of California, Berkeley 510.575.9515 www.plevin.com

…

On Jan 25, 2017, at 1:15 AM, Will Usher ***@***.***> wrote: Thanks for the feedback @rjplevin <https://github.com/rjplevin>. I think this could go one of two ways. The methods themselves tend to be quite involved, despite the uniform interface that SALib provides to each of them, and understanding the sensitivity indices obtained from the methods requires some level of understanding of the methods - for which the papers may be the best source. However, I'm definitely supportive of SALib becoming a not only easy to use, but also a source of new understanding about the methods and their similarities and differences. The question is how to go about this. As noted above, the interface to each of the methods is very similar - you could easily swap in Morris instead of Sobol in the example to which you provided a link. And cursory a description of the sensitivity indices is included in the docstrings of each of the methods. Could these be expanded to provide more information about interpretation? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#133 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADXRHg75cxH0EAsuxMDuvmyjQkiT6VA3ks5rVxK5gaJpZM4Ls74T>.

[willu47]

Hi @rjplevin, I'm sorry I haven't replied sooner. There are some detailed questions here, and I think the easiest way to answer them is to direct you toward Saltelli's book from 2008,

Saltelli, ., M Ratto, T., Andres, F., Campolongo, J., Cariboni, D., Gatelli, M., Saisana, and S. Tarantola. Global Sensitivity Analysis: The Primer. 1st ed. John Wiley & Sons Ltd, 2008. ISBN: 978-0-470-05997-5

This book provides a general overview of variance based sensitivity analyses. Section 6.5 (pg 272) outlines which methods to use when and how long they take to run.

A good overview paper is (full disclosure: Jim Hall is my boss):

Pianosi, Francesca, Keith Beven, Jim Freer, Jim W. Hall, Jonathan Rougier, David B Stephenson, and Thorsten Wagener. Sensitivity Analysis of Environmental Models: A Systematic Review with Practical Workflow. Environmental Modelling & Software 79 (May 2016): 214–32. doi:10.1016/j.envsoft.2016.02.008.

On the whole, I'm supportive of collecting many of your suggestions into the library documentation. We just need to work out how to do it!