Things that don't work in this project.

[Mt-Perazim]

First big thanks to the developer for the project! I noticed that a few things do not work and I hope to get help here.

So I tried this project today and right now its a bit frustrating.
I got it running on the HoloLens 2 and the python server is also running.

So what is my problem?

1. After the application started on the HoloLens and the connection to the python server is established, I can click "save spatial image" and it works! If I do so, the server recognize the command under if header == 'f': # save spatial camera images and saves it on under /data
2. If I click then on "save ahat" and then again on "save spatial image" the server get the command to trigger if header == 's': # save depth sensor images instead of getting 'f' again. So I gonna check the code in unity to find the reason for that.
3. Regardless of the order in which something is clicked, the AHAT command never works! Nothing happens on the python server side.
4. As already someone mentioned on the issue page, the stop button causes the application to crash.

• --Edit-- 10.Feb2022

So I tried to debug the unity part via adding text from the exception into the ui text, that already exists. I did not fix the problem but this is the current information:

1. ResearchModeVideoStream.cs:
   Clicking on the AHAT-Btn calls SaveAHATSensorDateEvent() . This method has the line var depthMap = researchMode.GetDepthMapBuffer(); which returns null and then passes depthMap (which is null) to the method SendUITunt16Async() in the next step.
2. TCPClient.cs:
   SendUITunt16Async() throws then a NullReferenceException :

System.NullReferenceException: Object reference not set to an instance of an object.
at TCPClient+<SendUINT16Async>d_16.MoveNext()[0x00000] in <000000000000000000000000000000000>:0
at System.Runtime.CompilerService.AsyncVoidMethodBuilder.Start[TStateMachine](TStateMachine&stateMachine)[0x00000] in <00000000000000000000000000000000000>:0
at ResyearchModeVideosStream.SaveAHAATSennsorDataEvent()[0x00000] in <000000000000000000000000000000000000000>:0
at System.Threading.threadStart.Invoke()[0x00000] in<000000000000000000000000000000000000000>:0
at UnityEngine.Events.UnitEvent.Invoke()[0x00000] in <000000000000000000000000000000000000000>:0
at Microsoft.MixedReality.Toolkit.UI.Interactable.SendOnClick

3. VS-Output:

Exception thrown at 0x00007FFB770F5E5C (KernelBase.dll) in HL2ResearchModeUnitySample.exe: 0x800706BA: The RPC server is unavailable.
onecore\com\combase\dcomrem\preventrundownbias.cpp(1310)\combase.dll!00007FFB769C61B0: (caller: 00007FFB769C606C) LogHr(1) tid(444) 800706BA The RPC server is unavailable.
Exception thrown at 0x00007FFB770F5E5C in HL2ResearchModeUnitySample.exe: Microsoft C++ exception: Il2CppExceptionWrapper at memory location 0x00000041099FE2A0.

• --Edit-- 16.Feb2022

After fixing the wrong depth-mode-issue I thought I could save the point cloud data to a .ply on the py-server, but the server cant handle the point cloud data! So I get the following Error:

--------------------------
Header: p
524293
Length of point cloud:262144
Oops! <class 'ValueError'> occurred.
Traceback (most recent call last):
  File "C:\Users\XXXX\source\repos\HoloLens2-ResearchMode-Unity-master\python\TCPServer.py", line 86, in tcp_server
    pointcloud_np = np.frombuffer(data[5:5 + n_pointcloud * 3 * 4], np.float32).reshape((-1, 3))
ValueError: cannot reshape array of size 131072 into shape (3)
Closing socket ...
Press any key ...

So this code cant reshape it because the content/length cannot be divided by 3. Why is this happening and what is wrong?

[Mt-Perazim]

does someone know why GetDepthMapBuffer returns null? @petergu684 @HoloAdventure can you please take a quick look at this issue?

[Mt-Perazim]

I found the solution.

Short: The wrong depth sensors where initialized and the c++-code had no error handling for that.

Long: The problem was that DepthSensorMode was serialized and visible in the unity editor. So depthSensorMode was in the editor view set to LongThrow (by default) and the whole code was expecting ShortThrow what you set at the beginning of the class ResearchModeVideoStream with [SerializeField] DepthSensorMode depthSensorMode = DepthSensorMode.ShortThrow;. So the depth sensor for long was initialized and SaveAHATSensorDataEvent() got for depthMap and AbImage = null.

[Mt-Perazim]

@petergu684 @HoloAdventure
So there is another problem with reshaping the point cloud data. I added the information to the first post!

[Mt-Perazim]

I fixed my reshape problem by realizing that var depthMap = researchMode.GetDepthMapBuffer(); is not the right method for that and that the python code is expecting a byte array, which was before a float array. And the method SendUINT16Async() with one parameter is also not the right method (eventhough the comments suggest that you should use this method) for that, so I coded another one.

    public void SavePointCloud()
    {
#if ENABLE_WINMD_SUPPORT
        var pointCloudBuffer = researchMode.GetPointCloudBuffer();
#if WINDOWS_UWP
        if (tcpClient != null)
        {
            tcpClient.SendSingleAsync(pointCloudBuffer );
        }
#endif
#endif
    }

    bool lastMessageSent = true;
    public async void SendSingleAsync(Single[] data)
    {
        if (!lastMessageSent) return;
        lastMessageSent = false;
        try
        {
            // Write header
            dw.WriteString("p"); // header "p" 

            // Write point cloud
            dw.WriteInt32(data.Length);
            dw.WriteBytes(SingleToBytes(data));

            // Send out
            await dw.StoreAsync();
            await dw.FlushAsync();
        }
        catch (Exception ex)
        {
            SocketErrorStatus webErrorStatus = SocketError.GetStatus(ex.GetBaseException().HResult);
            Debug.Log(webErrorStatus.ToString() != "Unknown" ? webErrorStatus.ToString() : ex.Message);
        }
        lastMessageSent = true;
    }

[Mt-Perazim]

I have reopened this issue for future visitors to see.

[agroenenberg]

@arsi0001 , thanks so much for your contribution! I have one more question, since I still got the problem that the point cloud is not received properly. I think I wrote the SingleToBytes(); function correctly, but I can't think of another place where the bug could be. How did you solve the problem? Here's my code:

byte[] SingleToBytes(float[] data)
{
    byte[] floatInBytes = new byte[data.Length * sizeof(float)];
    System.Buffer.BlockCopy(data, 0, floatInBytes, 0, floatInBytes.Length);
    return floatInBytes;
}

[Mt-Perazim]

@agroenenberg
Hey, your method should be fine, mine looked the same:

    byte[] SingleToBytes(Single[] data)
    {
        byte[] singleInBytes = new byte[data.Length * 4];
        System.Buffer.BlockCopy(data, 0, singleInBytes, 0, singleInBytes.Length);

        return singleInBytes;
    }

What is your error message? Or what's wrong with the point cloud you received?

[agroenenberg]

Hey @arsi0001, Solved it, overlooked a float conversion!

[Yuhan-Shen]

I found the solution.

Short: The wrong depth sensors where initialized and the c++-code had no error handling for that.

Long: The problem was that DepthSensorMode was serialized and visible in the unity editor. So depthSensorMode was in the editor view set to LongThrow (by default) and the whole code was expecting ShortThrow what you set at the beginning of the class ResearchModeVideoStream with [SerializeField] DepthSensorMode depthSensorMode = DepthSensorMode.ShortThrow;. So the depth sensor for long was initialized and SaveAHATSensorDataEvent() got for depthMap and AbImage = null.

Hi, I can use researchMode.InitializeLongDepthSensor(); researchMode.StartLongDepthSensorLoop(enablePointCloud); researchMode.GetLongDepthMapBuffer() to get the long_throw depth images. But after I set [SerializeField] DepthSensorMode depthSensorMode = DepthSensorMode.ShortThrow; I am unable to access the short_throw depth images by researchMode.InitializeDepthSensor(); researchMode.StartDepthSensorLoop(enablePointCloud); researchMode.GetDepthMapBuffer(). The error is the same as yours (var depthMap = researchMode.GetDepthMapBuffer(); returns null). Do you have any idea why this happens and how I can get the short_throw depth images? Thanks a lot!

[Mt-Perazim]

@Yuhan-Shen
If all lines in the code are set to shortThrow, it should work. Like I already said, I got that error only because there was a hardcoded line for a different depthSensoreMode.

Do you rely on the shortThrow? Because the shortThrow has a bug anyway, that is caused by the original researchmode project.

[Yuhan-Shen]

@Yuhan-Shen If all lines in the code are set to shortThrow, it should work. Like I already said, I got that error only because there was a hardcoded line for a different depthSensoreMode.

Do you rely on the shortThrow? Because the shortThrow has a bug anyway, that is caused by the original researchmode project.

Thank you! I really appreciate your reply.

To clarify, is the depthSensorMode set only in Unity/C#? Do I need to check the C++ plugins?

Also, can you please let me know what is the bug for shortThrow? Thanks!

[Mt-Perazim]

To clarify, is the depthSensorMode set only in Unity/C#? Do I need to check the C++ plugins?

No, no need to check there. The only thing missing in c++ is the handling for the case of wrong depth mode. So if you initialize for short but then ask for the long depth data.

Here is the bug.