sometimes GPU initialization is very slow

[mli]

hi guys,

probably it's due to my driver problem. i found sometimes it's extremely slow to get the message such as

"finish initialization with 2 devices"

for example, it takes >10 second to launch on GTX 970 with NVIDIA-SMI 343.19 and cuda 6.5. on another dual GTX 980 machine with NVIDIA-SMI 349.16, launching on the second GPU takes 5 second. but caffe seems has no problem on it.

but it works well on my other machines, such as single/dual/four k40 (gtx 980).

any idea?

best
mu

[antinucleon]

I am not sure what happened behind. I met this situation too. I use Ctrl +
C if it is more than 5 second, after kill it, the next running will be much
faster.

On Thu, Apr 23, 2015 at 3:26 PM Mu Li [email protected] wrote:

hi guys,

probably it's due to my driver problem. i found sometimes it's extremely
slow to get the message such as

"finish initialization with 2 devices"

for example, it takes >10 second to launch on GTX 970 with NVIDIA-SMI
343.19 and cuda 6.5. on another dual GTX 980 machine with NVIDIA-SMI
349.16, launching on the second GPU takes 5 second. but caffe seems has no
problem on it.

but it works well on my other machines, such as single/dual/four k40 (gtx
980).

any idea?

best
mu

—
Reply to this email directly or view it on GitHub
#108.

[jonbakerfish]

Have you encountered the slow starting of nvidia-smi problem? The reason for slow initialization might due to the driver persistence issue. NVIDIA's driver unloads a lot of stuff and releases state when the there is no client (X11 or an application). You can use # nvidia-smi -pm 1 to enable the persistent mode or use the persistence-daemon (preferred).

There might be some other reasons for slow starting:

• First CUDA call takes 13 seconds (Nouveau was the issue.)
• First CUDA function call very slow (more than a minute) on GTX 680 only (Compiling with "-gencode arch=compute_20,code=sm_20 -gencode arch=compute_30,code=sm_30" resolved the problem.)