verbs/man/mr: Describe the visible behavior of relaxed ordering

Detail the impact of using IBV_ACCESS_RELAXED_ORDERING on the API, as
it can change the visible ordering of transport message data placement.
This level of detail is required for developers to understand the impact
of setting the bit.

The formating of the memory registration flags is updated as a result
of this change, but only the relaxed ordering details are updated.

Signed-off-by: Sean Hefty <[email protected]>

libibverbs/man/ibv_reg_mr.3

@@ -34,27 +34,48 @@ The argument
 describes the desired memory protection attributes; it is either 0 or the bitwise OR of one or more of the following flags:
 .PP
 .TP
-.B IBV_ACCESS_LOCAL_WRITE \fR Enable Local Write Access
+.B IBV_ACCESS_LOCAL_WRITE \fR
+Enable Local Write Access
 .TP
-.B IBV_ACCESS_REMOTE_WRITE \fR Enable Remote Write Access
+.B IBV_ACCESS_REMOTE_WRITE \fR
+Enable Remote Write Access
 .TP
-.B IBV_ACCESS_REMOTE_READ\fR Enable Remote Read Access
+.B IBV_ACCESS_REMOTE_READ\fR
+Enable Remote Read Access
 .TP
-.B IBV_ACCESS_REMOTE_ATOMIC\fR Enable Remote Atomic Operation Access (if supported)
+.B IBV_ACCESS_REMOTE_ATOMIC\fR
+Enable Remote Atomic Operation Access (if supported)
 .TP
-.B IBV_ACCESS_FLUSH_GLOBAL\fR Enable Remote Flush Operation with global visibility placement type (if supported)
+.B IBV_ACCESS_FLUSH_GLOBAL\fR
+Enable Remote Flush Operation with global visibility placement type (if supported)
 .TP
-.B IBV_ACCESS_FLUSH_PERSISTENT\fR Enable Remote Flush Operation with persistence placement type (if supported)
+.B IBV_ACCESS_FLUSH_PERSISTENT\fR
+Enable Remote Flush Operation with persistence placement type (if supported)
 .TP
-.B IBV_ACCESS_MW_BIND\fR Enable Memory Window Binding
+.B IBV_ACCESS_MW_BIND\fR
+Enable Memory Window Binding
 .TP
-.B IBV_ACCESS_ZERO_BASED\fR Use byte offset from beginning of MR to access this MR, instead of a pointer address
+.B IBV_ACCESS_ZERO_BASED\fR
+Use byte offset from beginning of MR to access this MR, instead of a pointer address
 .TP
-.B IBV_ACCESS_ON_DEMAND\fR Create an on-demand paging MR
+.B IBV_ACCESS_ON_DEMAND\fR
+Create an on-demand paging MR
 .TP
-.B IBV_ACCESS_HUGETLB\fR Huge pages are guaranteed to be used for this MR, applicable with IBV_ACCESS_ON_DEMAND in explicit mode only
+.B IBV_ACCESS_HUGETLB\fR
+Huge pages are guaranteed to be used for this MR, applicable with IBV_ACCESS_ON_DEMAND in explicit mode only
 .TP
-.B IBV_ACCESS_RELAXED_ORDERING\fR Allow system to reorder accesses to the MR to improve performance
+.B IBV_ACCESS_RELAXED_ORDERING\fR
+This setting allows the NIC to relax the order that data is transfered between the
+network and the target memory region. Relaxed ordering allows network initiated
+writes (such as incoming message send or RDMA write operations) to reach memory
+in an arbitrary order. This can improve the performance of some applications.
+However, relaxed ordering has the following impact: RDMA write-after-write message
+order is no longer guaranteed. (Send messages will still match posted receive buffers
+in order.) Back-to-back network writes that target the same memory region
+leave the region in an unknown state. Relaxed ordering does not change completion
+semantics, such as data visibility. That is, a completion still ensures that all
+data is visible, including data from prior transfers. Relaxed ordered operations
+will also not bypass atomic operations.
 .PP
 If
 .B IBV_ACCESS_REMOTE_WRITE