Gb/multi gpu #138
Gb/multi gpu #138
Conversation
|
@bnb32 can you take a look at what might be going on with the data-centric test? I have no idea how my edits would affect that code https://github.com/NREL/sup3r/actions/runs/3876667898/jobs/6610776941 |
| training_weights, | ||
| device_name=f'/gpu:{i}', | ||
| **calc_loss_kwargs)) | ||
| for i, future in enumerate(futures): |
There was a problem hiding this comment.
why don't we need as_completed() here?
There was a problem hiding this comment.
Also, applying the different gradients like this is better than applying the average of the gradients once?
There was a problem hiding this comment.
we could use as_completed() but the futures will finish at basically the same time so i was just keeping it simple. And i dont think either is better - averaging with a single weight update might actually be more intuitive, but this is nice and simple.
| @@ -960,3 +1113,54 @@ def _tf_generate_wind(self, low_res, hi_res_topo): | |||
| raise RuntimeError(msg) from e | |||
|
|
|||
| return hi_res | |||
|
|
|||
| @tf.function() | |||
There was a problem hiding this comment.
having this copied code hurts me. but alas...
|
|
||
| return grad, loss_details | ||
|
|
||
| def run_gradient_descent(self, low_res, hi_res_true, training_weights, |
There was a problem hiding this comment.
this is a nice simple solution. well done!
|
Wish we could restructure to avoid duplicating some code but overall an elegant solution to the multi-gpu problem. |
… multi_gpu flag to train entry point
grantbuster
force-pushed
the
gb/multi_gpu
branch
from
3680e90
to
d6e2cd1
Compare
No description provided.