Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: hypre improvements #3339

Draft
wants to merge 29 commits into
base: develop
Choose a base branch
from
Draft

Conversation

victorapm
Copy link
Contributor

@victorapm victorapm commented Sep 9, 2024

  • Replace internal GEOS SDC implementation with hypre's
  • Remove tracking of certain SDC times
  • Add hypre log level support
  • Update hypre umpire pools naming
  • Add missing coarse solver options in AMG
  • Remove unused SDC variables in GEOS data structures
  • Add separateComponents option to mechanics solver setup in MGR
  • Move addCommaSeparators to StringUtilities
  • Add unknowns and nonzeros info to linear solver message, e.g.:
Linear Solver | Success | Unknowns: 645,162 | Nonzeros: 42,898,950 | Iterations: 39 | Final Rel Res: 9.4042e-05 | Setup Time: 0.538 s | Solve Time: 0.933 s

Requires GEOS-DEV/thirdPartyLibs#286

@victorapm victorapm self-assigned this Sep 9, 2024
@ryar9534
Copy link
Contributor

ryar9534 commented Sep 10, 2024

Here is a quick test I ran on a version of SPE10 with burdens, compositional poromechanics, though with homogeneous properties, (so its just a high perm slab between two low perm boxes). Should be about 40 million DoF, and I ran on Dane.

Here were the results with develop a few weeks ago

Mpi ranks Total Run Time (s) Linear Solver Setup (s) Linear Solver Solve (s) Total GMRES Iterations (always 43 Total Newton iters)
64 2087 523 1966 1850
128 1451 343 1381 1833
256 963 234 662 1890
512 624 182 404 1974

Here are the results with the new Hypre

Mpi ranks Total Run Time (s) Linear Solver Setup (s) Linear Solver Solve (s) Total GMRES Iterations (always 43 Total Newton iters)
64 1568 233 1162 2281
128 1115 139 865 2229
256 723 85 572 2293
512 470 85 347 2366

Note that this does not double the number of nodes for each test, I will do that now

@ryar9534
Copy link
Contributor

ryar9534 commented Sep 11, 2024

Old results for heterogeneous SPE10 compositional poromechanics on CPU (dane)

Dane cores (Mpi ranks) Total Run Time (s) Linear Solver Setup (s) Linear Solver Solve (s) Time steps/Nonlin iters / Lin iters
2 (224) 2409 488 1764 24/76/3464
4 (448) 1440 380 977 24/76/3862
8 (896) 1075 447 583 24/76/4286

New results for heterogeneous SPE10 compositional poromechanics on CPU (dane)

Dane cores (Mpi ranks) Total Run Time (s) Linear Solver Setup (s) Linear Solver Solve (s) Time steps/Nonlin iters / Lin iters
2 (224) 1770 185 1425 24/76/3882
4 (448) 952 152 719 24/76/3960
8 (896) 686 225 417 24/76/4228

@victorapm victorapm marked this pull request as ready for review October 4, 2024 20:12
HYPRE_SetUmpirePinnedPoolName( "HYPRE_PINNED" );
#endif

HYPRE_SetLogLevel( getenv( "HYPRE_LOG_LEVEL" ) ? atoi( getenv( "HYPRE_LOG_LEVEL" ) ) : 0 );
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can it be set by linear solver log level?

Copy link
Contributor Author

@victorapm victorapm Oct 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, but I did this way because the information provided here is mainly for developers. Other libraries such as umpire and rocsparse work this way as well (using env variable). I am open to change the strategy here if needeed

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sounds reasonable, thanks, no need to change

@@ -80,7 +80,6 @@ HydrofractureSolver< POROMECHANICS_SOLVER >::HydrofractureSolver( const string &
// This may need to be different depending on whether poroelasticity is on or not.
m_linearSolverParameters.get().mgr.strategy = LinearSolverParameters::MGR::StrategyType::hydrofracture;
m_linearSolverParameters.get().mgr.separateComponents = false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@victorapm how about this one?

Copy link
Contributor Author

@victorapm victorapm Oct 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I need to check this one with @rrsettgast @castelletto1 and @CusiniM, but I guess SDC for the fracture displacements is not very useful so I left it false. I can be wrong though

@victorapm victorapm marked this pull request as draft October 10, 2024 21:27
@victorapm
Copy link
Contributor Author

We are investigating a few convergence issues with Poromechanics runs using this branch. @drmichaeltcvx, please you can put your tests on hold until the issue is sorted out

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants