Test AD on more algorithms

[lxvm]

Goal is to address #150 and #57.
TODO:

• Loop through all algorithms in AD tests
• Add FastGaussQuadrature to AD tests
• Fix any bugs that come up

[lxvm]

@ChrisRackauckas This pr should be good to go! The test suite is updated to test various algorithms for the various problem types in the library on various kinds of inputs. There were some issues in Zygote that I made upstream prs to fix that would simplify the test suite, but I managed to write differentiable integrands.

Also, I didn't add the monte carlo algorithms to these tests because the randomness slows the convergence, and perhaps the primal and dual computations should be done in the same library call to ensure that the same quadrature is used, which I believe is also an approach StochasticAD uses.

[ChrisRackauckas]

This looks great!

Also, I didn't add the monte carlo algorithms to these tests because the randomness slows the convergence, and perhaps the primal and dual computations should be done in the same library call to ensure that the same quadrature is used, which I believe is also an approach StochasticAD uses.

What's the issue though with CubaCuhre? I'm a bit surprised to see that one lumped in there.

Indeed, when the algorithm can be differentiated through, I think it is more stable to have it use the same points so we should probably have a trait that effectively passes the duals along into those algorithms. Those will be only the pure Julia ones though, which is a rather small subset that we can opt-in.

[lxvm]

I didn't pay attention to CubaCuhre specifically, but there was a Cuba algorithm tested that didn't pass the convergence test comparing the finite differences to ForwardDiff gradients that requests a tolerance of 3 digits. Thinking about it, finite differences on the MC algorithms is even worse since each integral evaluation has its own random seed, and the same is true of comparing different ADs. Anyway, that can be address in a follow-up pr.

Regarding the passing of duals into the integrand in the forward pass, it is already implemented and assumes that the library supports vector-valued integrands.