-
Notifications
You must be signed in to change notification settings - Fork 652
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
mana: Microsoft Azure Network Adapter (MANA) RDMA provider
Introduce a provider that exposes MANA devices to user applications. Signed-off-by: Long Li <[email protected]>
- Loading branch information
1 parent
69056ae
commit 443f196
Showing
25 changed files
with
1,480 additions
and
7 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -124,6 +124,11 @@ M: Sean Hefty <[email protected]> | |
| S: Supported | ||
| F: librdmacm/ | ||
|
|
||
| MANA USERSPACE PROVIDER (for mana_ib.ko) | ||
| M: Long Li <[email protected]> | ||
| S: Supported | ||
| F: providers/mana/ | ||
|
|
||
| MLX4 USERSPACE PROVIDER (for mlx4_ib.ko) | ||
| M: Yishai Hadas <[email protected]> | ||
| H: Roland Dreier <[email protected]> | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,5 +1,6 @@ | ||
| etc/libibverbs.d/ | ||
| usr/lib/*/libefa.so.* | ||
| usr/lib/*/libibverbs/lib*-rdmav*.so | ||
| usr/lib/*/libmana.so* | ||
| usr/lib/*/libmlx4.so.* | ||
| usr/lib/*/libmlx5.so.* |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,2 +1,2 @@ | ||
| # libefa, libmlx4 and libmlx5 are ibverbs provider that provides more functions. | ||
| ibverbs-providers: package-name-doesnt-match-sonames libefa1 libmlx4-1 libmlx5-1 | ||
| # libefa, libmana, libmlx4 and libmlx5 are ibverbs provider that provides more functions. | ||
| ibverbs-providers: package-name-doesnt-match-sonames libefa1 libmana1 libmlx4-1 libmlx5-1 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,13 @@ | ||
| rdma_shared_provider(mana libmana.map | ||
| 1 1.0.${PACKAGE_VERSION} | ||
| mana.c | ||
| manadv.c | ||
| qp.c | ||
| wq.c | ||
| ) | ||
|
|
||
| publish_headers(infiniband | ||
| manadv.h | ||
| ) | ||
|
|
||
| rdma_pkg_config("mana" "libibverbs" "${CMAKE_THREAD_LIBS_INIT}") |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,8 @@ | ||
| /* Export symbols should be added below according to | ||
| Documentation/versioning.md document. */ | ||
| MANA_1.0 { | ||
| global: | ||
| manadv_set_context_attr; | ||
| manadv_init_obj; | ||
| local: *; | ||
| }; |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,5 @@ | ||
| rdma_man_pages( | ||
| manadv.7.md | ||
| manadv_init_obj.3.md | ||
| manadv_set_context_attr.3.md | ||
| ) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,47 @@ | ||
| --- | ||
| layout: page | ||
| title: MANADV | ||
| section: 7 | ||
| tagline: Verbs | ||
| date: 2022-05-16 | ||
| header: "MANA Direct Verbs Manual" | ||
| footer: mana | ||
| --- | ||
|
|
||
| # NAME | ||
| manadv - Direct verbs for mana devices | ||
|
|
||
| This provides low level access to mana devices to perform direct operations, | ||
| without general branching performed by libibverbs. | ||
|
|
||
| # DESCRIPTION | ||
| The libibverbs API is an abstract one. It is agnostic to any underlying | ||
| provider specific implementation. While this abstraction has the advantage | ||
| of user applications portability, it has a performance penalty. For some | ||
| applications optimizing performance is more important than portability. | ||
|
|
||
| The mana direct verbs API is intended for such applications. | ||
| It exposes mana specific low level operations, allowing the application | ||
| to bypass the libibverbs API. | ||
|
|
||
| This version of the driver supports one QP type: IBV_QPT_RAW_PACKET. To use | ||
| this QP type, the application is required to use manadv_set_context_attr() | ||
| to set external buffer allocators for allocating queues, and use | ||
| manadv_init_obj() to obtain all the queue information. The application | ||
| implements its own queue operations, bypassing libibverbs API for | ||
| sending/receiving traffic over the queues. At hardware layer, IBV_QPT_RAW_PACKET | ||
| QP shares the same hardware resource as the Ethernet port used in the kernel. | ||
| The software checks for exclusive use of the hardware Ethernet port, and will | ||
| fail the QP creation if the port is already in use. To create a | ||
| IBV_QPT_RAW_PACKET on a specified port, the user needs to configure the system | ||
| in such a way that this port is not used by any other software (including the | ||
| Kernel). If the port is used, ibv_create_qp() will fail with errno set to EBUSY. | ||
|
|
||
| The direct include of manadv.h together with linkage to mana library will | ||
| allow usage of this new interface. | ||
|
|
||
| # SEE ALSO | ||
| **verbs**(7) | ||
|
|
||
| # AUTHORS | ||
| Long Li <[email protected]> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,83 @@ | ||
| --- | ||
| layout: page | ||
| title: manadv_init_obj | ||
| section: 3 | ||
| tagline: Verbs | ||
| --- | ||
|
|
||
| # NAME | ||
| manadv_init_obj \- Initialize mana direct verbs object from ibv_xxx structures | ||
|
|
||
| # SYNOPSIS" | ||
| ```c | ||
| #include <infiniband/manadv.h> | ||
|
|
||
| int manadv_init_obj(struct manadv_obj *obj, uint64_t obj_type); | ||
| ``` | ||
| # DESCRIPTION | ||
| manadv_init_obj() | ||
| This function will initialize manadv_xxx structs based on supplied type. The information | ||
| for initialization is taken from ibv_xx structs supplied as part of input. | ||
| # ARGUMENTS | ||
| *obj* | ||
| : The manadv_xxx structs be to returned. | ||
| ```c | ||
| struct manadv_qp { | ||
| void *sq_buf; | ||
| uint32_t sq_count; | ||
| uint32_t sq_size; | ||
| uint32_t sq_id; | ||
| uint32_t tx_vp_offset; | ||
| void *db_page; | ||
| }; | ||
| struct manadv_cq { | ||
| void *buf; | ||
| uint32_t count; | ||
| uint32_t cq_id; | ||
| }; | ||
| struct manadv_rwq { | ||
| void *buf; | ||
| uint32_t count; | ||
| uint32_t size; | ||
| uint32_t wq_id; | ||
| void *db_page; | ||
| }; | ||
| struct manadv_obj { | ||
| struct { | ||
| struct ibv_qp *in; | ||
| struct manadv_qp *out; | ||
| } qp; | ||
| struct { | ||
| struct ibv_cq *in; | ||
| struct manadv_cq *out; | ||
| } cq; | ||
| struct { | ||
| struct ibv_wq *in; | ||
| struct manadv_rwq *out; | ||
| } rwq; | ||
| }; | ||
| ``` | ||
|
|
||
| *obj_type* | ||
| : The types of the manadv_xxx structs to be returned. | ||
|
|
||
| ```c | ||
| enum manadv_obj_type { | ||
| MANADV_OBJ_QP = 1 << 0, | ||
| MANADV_OBJ_CQ = 1 << 1, | ||
| MANADV_OBJ_RWQ = 1 << 2, | ||
| }; | ||
| ``` | ||
| # RETURN VALUE | ||
| 0 on success or the value of errno on failure (which indicates the failure reason). | ||
|
|
||
| # AUTHORS | ||
| Long Li <[email protected]> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,65 @@ | ||
| --- | ||
| layout: page | ||
| title: manadv_set_context_attr | ||
| section: 3 | ||
| tagline: Verbs | ||
| --- | ||
|
|
||
| # NAME | ||
| manadv_set_context_attr - Set context attributes | ||
|
|
||
| # SYNOPSIS | ||
| ```c | ||
| #include <infiniband/manadv.h> | ||
|
|
||
| int manadv_set_context_attr(struct ibv_context *context, | ||
| enum manadv_set_ctx_attr_type attr_type, | ||
| void *attr); | ||
| ``` | ||
| # DESCRIPTION | ||
| manadv_set_context_attr gives the ability to set vendor specific attributes on | ||
| the RDMA context. | ||
| # ARGUMENTS | ||
| *context* | ||
| : RDMA device context to work on. | ||
| *attr_type* | ||
| : The type of the provided attribute. | ||
| *attr* | ||
| : Pointer to the attribute to be set. | ||
| ## attr_type | ||
| ```c | ||
| enum manadv_set_ctx_attr_type { | ||
| /* Attribute type uint8_t */ | ||
| MANADV_SET_CTX_ATTR_BUF_ALLOCATORS = 0, | ||
| }; | ||
| ``` | ||
| *MANADV_SET_CTX_ATTR_BUF_ALLOCATORS* | ||
| : Provide an external buffer allocator | ||
|
|
||
| ```c | ||
| struct manadv_ctx_allocators { | ||
| void *(*alloc)(size_t size, void *priv_data); | ||
| void (*free)(void *ptr, void *priv_data); | ||
| void *data; | ||
| }; | ||
| ``` | ||
| *alloc* | ||
| : Function used for buffer allocation instead of libmana internal method | ||
|
|
||
| *free* | ||
| : Function used to free buffers allocated by alloc function | ||
|
|
||
| *data* | ||
| : Metadata that can be used by alloc and free functions | ||
|
|
||
| # RETURN VALUE | ||
| Returns 0 on success, or the value of errno on failure | ||
| (which indicates the failure reason). | ||
|
|
||
| # AUTHOR | ||
| Long Li <[email protected]> |
Oops, something went wrong.