Fix issue when setting memory growth

[marcelvanworkum]

When calling tf.config.experimental.set_memory_growth, we want to set the value to True for all physical devices.

Otherwise, the following ValueError is produced: 'Memory growth cannot differ between GPU devices'

I ran into this issue when running python detect.py --weights ./checkpoints/yolov3-tiny.tf --tiny --image ./data/street.jpg on a device with two RTX 2080TI.

raise ValueError("Memory growth cannot differ between GPU devices")
 ValueError: Memory growth cannot differ between GPU devices

[marcelvanworkum]

Related to #217, although I did not have any issues running convert.py with multiple GPUs.

My abbreviated log when running python detect.py --weights ./checkpoints/yolov3-tiny.tf --tiny --image ./data/street.jpg is:

2020-04-06 12:07:58.189200: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:1a:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2020-04-06 12:07:58.189703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties:
pciBusID: 0000:68:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
...
2020-04-06 12:07:58.199253: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1
...
2020-04-06 12:07:58.424402: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 0 with properties:
pciBusID: 0000:1a:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
2020-04-06 12:07:58.425695: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1555] Found device 1 with properties:
pciBusID: 0000:68:00.0 name: GeForce RTX 2080 Ti computeCapability: 7.5
coreClock: 1.545GHz coreCount: 68 deviceMemorySize: 10.76GiB deviceMemoryBandwidth: 573.69GiB/s
...
2020-04-06 12:07:58.430705: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1697] Adding visible gpu devices: 0, 1
...
2020-04-06 12:07:58.438303: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost
/replica:0/task:0/device:GPU:0 with 10311 MB memory) -> physical GPU (device: 0, name: GeForce RTX 2080 Ti, pci bus id: 0000:1
a:00.0, compute capability: 7.5)
2020-04-06 12:07:58.439735: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1241] Created TensorFlow device (/job:localhost
/replica:0/task:0/device:GPU:1 with 10272 MB memory) -> physical GPU (device: 1, name: GeForce RTX 2080 Ti, pci bus id: 0000:6
8:00.0, compute capability: 7.5)
...
I0406 12:08:04.025114 140135716407040 convert.py:26] sanity check passed
I0406 12:08:04.551327 140135716407040 convert.py:29] weights saved