Poor Performance in TensorFlow 2.1

[kykosic]

I was refactoring the "American_Option_Black_Sholes.ipynb" to use the new fd_solvers API with TensorFlow 2.1, but it has much worse performance than in the previous version that was pre-TF2.0.

Here is the notebook re-written in eager style and ran on the colab GPU: https://colab.research.google.com/drive/1FVYloxIrSvlSrp1aMPLxN4f4KlvPI8l0

As you can see the CPU and GPU options/sec benchmarks are both around 1/3 what they were in the old notebook saved on the master branch. I've also tried running the same code, but using the tf.compat.v1 Session and graph; this did not improve performance at all. I've also tried to profile the code using tf.summary.trace_on(graph=True, profiler=True), but it doesn't produce any useful information.

Is this performance something I'm doing wrong with my implementation, is there perhaps some better way I can profile the core library for issues, or is this related to some performance issues with TF2.0 itself?

[cyrilchim]

Hi Kyle!

Thanks for putting your efforts to make the example TF2-friendly. We will add benchmarking tools later.

As for now here are some remarks:

1. I guess, you are running the colab with a public GPU which is shared. Sometimes it will run faster, sometimes slower. I just run your colab and it has executed twice as fast on a GPU. As an alternative, you can get some free GCP credits and try running the colab through AI Platform Notebooks (create a VM instance with a GPU and customize it to Tesla V100).

2. Try running your computations twice. You should observe the second run much faster than the first one. This is because TF has to actually "compile" the graph in the first run (see, e.g., here or here). In the second run, if you do not change the shapes of the inputs (but feel free to change the values), no "compilation" is done, so the performance should be much better. As a practical example, say, you have 5 million options to price. You could price a batch of 5000 options at a time and use python for-loop to go through the whole portfolio.

Hope this helps!

[kykosic]

Thanks for the response @cyrilchim. I've since tested the code on a V100 with a warmup computation and saw significant performance improvements over the old API version.

I did still see slightly worse performance on the CPU variation, but I'm also having issues getting TF2.1 compiled with MKL on my instance so could be a cause.

Would a pull request with an update to that notebook be appropriate? Or are you planning on waiting for some more benchmarks that you mentioned?

[cyrilchim]

Hi Kyle,
The change looks very good. Please feel free to push it. Thank you!