-
Notifications
You must be signed in to change notification settings - Fork 280
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
mpiexec runs much slower than OpenMPI in Github Actions #6037
Comments
I have two tips:
|
Could you please help me with an example for creating the xml file? |
to create the xml file. For reference: https://linux.die.net/man/1/lstopo
I think OpenMPI only discovers hardware in |
The HWLOC_XMLFILE approach does not improve. I am still getting a similar long time. |
Could you try both ch3 and ch4 devices and see if there is any difference in timing? |
I used
|
Did you somehow hide the lines such as |
I re-ran to build MPICH with ch4, which took 31 minutes. |
It also runs very fast on my local redhat machine. |
Do you have a link to the recent github action test log? |
See below. You can also check their yml files for the configure options I used. |
Somehow every test seems ran twice:
|
This is because configure option What puzzles me is why the same configure settings used in both |
Even OpenMPI's 15 min is a big puzzle if it only run on local computer for less than a minute. That's 15x, a much bigger puzzle. Any ideas? |
I believe it is because github actions are running on a virtual environment. |
I believe virtualization nowadays are pretty good, i.e. I would think it won't slow down by more than 2x. Looking at the compilation time, it seems reasonably fast. |
If that is the case, then I have no idea why github actions is slower. |
Is there a way to show timestamp of each test? |
Yes. On the top-right corner, click the gear icon and click show timestamps and full screen. |
Just catching up here. Github Actions runners only have 2 virtual cores. The oversubscription might be really hurting. Could you test using the |
Also I guess I should try to clarify, why do we think that |
The trick of |
Thanks for confirming. This is another piece of evidence supporting adding a configuration in ch4 that can run without busy-polling. Whether that is ch4:sock or something else. |
I will be happy to do some profiling. Let me know. |
Is there now another combination with cc: @vicentebolea |
No. It is still sitting on our TODO list. |
Ok, thanks, just checking. |
I run two github action workflows for PnetCDF. One is using MPICH and the
other OpenMPI. They both run many small 4-process jobs (command: make ptest)
and all MPI jobs run on the same local host node. The MPICH version is always much
slower than OpenMPI. The times of the latest runs show
I have also tried to build MPICH 4.0.2 from source as part of workflow, but got a similar
timing results.
Is there a way to speed up mpiexec?
The text was updated successfully, but these errors were encountered: