UCT/IB/MLX5: suggest user to increase PF_LOG_BAR_SIZE when UAR allocation fails

[michal-shalev]

What ?

Improve error logging for UAR allocation failure in uct_ib_mlx5_devx_alloc_uar.

Why ?

To inform users to increase PF_LOG_BAR_SIZE and reboot when UAR allocation fails due to insufficient memory.
see also: open-mpi/ompi#6854

How ?

Update the error message in uct_ib_mlx5_devx_alloc_uar to suggest increasing PF_LOG_BAR_SIZE and mention the reboot requirement.

[yosefe]

1. "Consider increasing PF_LOG_BAR_SIZE using mlxconfig tool",
2. should we also check errno to have more specific error condition?
3. i think there could be other places that may fail because of the same reason, for example "mlx5dv_devx_obj_create(CQ)"

[michal-shalev]

1. added the "mlxconfig tool" to the string.
2. do you mean check "ENOMEM"?
3. added this, checking for other places

[yosefe]

2. Yes

[michal-shalev]

1. As you mentioned here: openmpi-3.1.1 np=256 ppn=128 UCX_TLS=rc fail open-mpi/ompi#6854 this requires a reboot so I added it to the string as well.
2. It looks like mlx5dv_devx_alloc_uar -> _mlx5dv_devx_alloc_uar -> mlx5_attach_dedicated_uar -> mlx5_alloc_dyn_uar -> execute_ioctl -> ioctl
   it returns the errno but I don't know if we can know for sure that it'll set ENOMEM when PF_LOG_BAR_SIZE is insufficient.
3. mlx5dv_devx_obj_create(CQ) doesn't fail because of PF_LOG_BAR_SIZE, in the email "
   After applying the PF_LOG_BAR_SIZE parameter update".
   I removed the commit for it.

Regarding non-DEVX flows
uct_worker_tl_data_get -> uct_ib_mlx5_iface_get_res_domain -> uct_ib_mlx5_res_domain_init -> ibv_alloc_td -> mlx5_alloc_td -> mlx5_attach_dedicated_uar-> mlx5_alloc_dyn_uar -> execute_ioctl -> ioctl
If that fails due to insufficient PF_LOG_BAR_SIZE, uct_ib_mlx5_res_domain_init function does not fail. Instead, it returns UCS_OK and logs a debug message.
The fallback mechanism uses the PD directly from the MD structure. This ensures continued operation, though with potentially reduced performance, especially for users sending a high volume of messages to different destinations.
This is because the shared UAR may not handle the high message rate as efficiently.

I can add this if ibv_alloc_td fails in uct_ib_mlx5_res_domain_init:
ucs_log(log_level, "Consider increasing PF_LOG_BAR_SIZE using mlxconfig tool");
However, this log might not be noticed because the entire flow does not fail.

Another suggestion is to change the log level from ucs_debug to ucs_warn:
ucs_warn("ibv_alloc_td() on %s failed: %m. Falling back to using the PD directly from the MD structure, which may reduce performance.", uct_ib_device_name(&md->dev))
This would make the suggestion to increase PF_LOG_BAR_SIZE more noticeable and highlight the potential performance impact.

[gleon99]

@michal-shalev per our offline discussion,

• (2) ENOMEM - please check whether it indeed returns.
• Other flows - IMO generally we shouldn't change the behavior in this PR. Flows should not turn from failing to succeeding as a result of this PR, neither logging levels should change. Only add a suggestion for the user, in the already existing log level.

[michal-shalev]

@gleon99 I'll write a test that calls mlx5dv_devx_alloc_uar until it fails and checks the errno to make sure what it returns exactly.
Other flows - I'll add a suggestion.