Put not supported using vineyard.RPCClient.

[dv-rastogi]

Describe your problem

How to put and get numpy arrays using vineyard.RPCClient.
As per the documentation, vineyard.RPCClient does have get and put, although when I try to invoke the methods, I get an error.

---

If is is a bug report, to help us reproducing this bug, please provide information below:

1. Your Operation System version (uname -a): Linux 5.10.0-22-cloud-amd64 # 1 SMP Debian 5.10.178-3 (2023-04-22) x86_64 GNU/Linux
2. The version of vineyard you use (vineyard.__version__): 0.14.5
3. Versions of crucial packages, such as gcc, numpy, pandas, etc.: numpy==1.24.3
4. Full stack of the error (if there are a crash):

Traceback (most recent call last):
  File "/home/divyansh19464/vin-write.py", line 21, in <module>
    payload_id = vineyard_rpc_client.put(payload)
  File "/home/divyansh19464/.local/lib/python3.9/site-packages/vineyard/core/builder.py", line 182, in put
    meta = get_current_builders().run(client, value, **kwargs)
  File "/home/divyansh19464/.local/lib/python3.9/site-packages/vineyard/core/builder.py", line 88, in run
    return self.__factory[ty](client, value, **kw)
  File "/home/divyansh19464/.local/lib/python3.9/site-packages/vineyard/data/tensor.py", line 65, in numpy_ndarray_builder
    meta.add_member('buffer_', build_numpy_buffer(client, value))
  File "/home/divyansh19464/.local/lib/python3.9/site-packages/vineyard/data/utils.py", line 138, in build_numpy_buffer
    return build_buffer(client, address, array.nbytes)
  File "/home/divyansh19464/.local/lib/python3.9/site-packages/vineyard/data/utils.py", line 125, in build_buffer
    existing = client.find_shared_memory(address)
AttributeError: 'vineyard._C.RPCClient' object has no attribute 'find_shared_memory'

5. Minimized code to reproduce the error:

vineyard_rpc_client = vineyard.connect("10.128.0.8", 9600)
payload = np.random.random((3, 2))
payload_id = vineyard_rpc_client.put(payload)
print(vineyard_rpc_client.get(payload_id))

[sighingnow]

Currently RPC client is designed for inspecting metadata of global/remote objects (see also: https://v6d.io/notes/key-concepts/objects.html), e.g.,

• when we have a global object and want to know metadata of a remote chunk, we need rpc_client.get()
• when we have many local chunks (we have the object ID for each chunk), we can create a new global object (global object cannot has direct blob member) using the rpc_client.put().

The error in the description above is expected behavior. The error message itself is a bit misleading and worth a fix.

[dv-rastogi]

I see, my use case is to put a numpy array from machine A to machine B's vineyard server. I had the following strategies in mind:

1. Start a vineyard server on machine A and do object migration
2. Use remote blob builder on top of the underlying numpy array buffer on machine A to build a remote blob on machine B, store the array metadata (shape) in a redis instance, and then query it from machine B.

Keeping solely performance in mind, do you have any suggestions?

[sighingnow]

Those two approaches shouldn't have big performance differences as they share the same codepath.

• for method 1, you will need vineyardd in machine A
• for method 2, vineyardd in machine A is not needed but you need a bit extra work to put the metadata. That should be ok as well if your object metadata is not every complicated.

[dv-rastogi]

Got it, that helps! Thank you.

[dv-rastogi]

Furthermore, I was not able to find much documentation on Object Migration and the APIs for the same. Where can I find that?