Monitored items and UA_VARIANT_DATA_NODELETE

[basyskom-jvoe]

When a data source read callback sets the storage type of the UA_Variant to UA_VARIANT_DATA_NODELETE and the data pointer refers to a fixed memory position (e.g. static variable), any monitored item for a variable node with such a data source callback doesn't detect any data changes.

If I remember correctly, there were changes to the behavior of variant copying with UA_VARIANT_DATA_NODELETE some time ago. Would these changes explain the observed behavior and if so, is there a way to deal with it?

Checklist

Please provide the following information:

• open62541 Version (release number or git tag): v1.4.1

[jpfr]

That is strange.

what could happen is that we keep the original variant around for the comparison.
So when we compare the old variant still points to the static memory that now contains the new value.

[jpfr]

Added a unit test for this in #6518.
It appears to work correctly.

[basyskom-jvoe]

This is indeed very strange...

I have added debug output to detectValueChange():

    UA_LOG_INFO_SUBSCRIPTION(server->config.logging, mon->subscription, "XXXXXXXXXXXXXXXXXXXXXX Monitored item %p vs. %p", mon->lastValue.value.data, dv->value.data);

and according to this output, the data pointer is indeed identical for the old and new variants:

Session "urn:rigel:UnifiedAutomation:UaExpert"	| Subscription 3 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0xdab908 vs. 0xdab908

EDIT: My callback sets the server timestamp in the DataValue and I can get it working if I modify the monitored item in UaExpert to trigger on StatusValueTimestamp. Could this also be the reason why your test works?

EDIT 2:
I have built the current 1.4 branch with your test on my desktop. The same debug output is quite unexpected, I would have expected the pointer address for dv->value.data to be always the same, but it changes:

[1601-01-01 00:02:35.290 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item (nil) vs. 0x5fdd80aa51b0
[1601-01-01 00:02:35.390 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa51b0 vs. 0x5fdd80aa51d0
[1601-01-01 00:02:35.490 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa51d0 vs. 0x5fdd80aa51f0
[1601-01-01 00:02:35.590 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa51f0 vs. 0x5fdd80aa5210
[1601-01-01 00:02:35.690 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa5210 vs. 0x5fdd80aa51f0
[1601-01-01 00:02:35.790 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa51f0 vs. 0x5fdd80aa5210
[1601-01-01 00:02:35.890 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa5210 vs. 0x5fdd80aa51f0
[1601-01-01 00:02:35.990 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa51f0 vs. 0x5fdd80aa5210
[1601-01-01 00:02:36.090 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa5210 vs. 0x5fdd80aa51f0
[1601-01-01 00:02:36.190 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa51f0 vs. 0x5fdd80aa5210
[1601-01-01 00:02:36.290 (UTC+0000)] info/server	Subscription 0 | XXXXXXXXXXXXXXXXXXXXXX Monitored item 0x5fdd80aa5210 vs. 0x5fdd80aa51f0

[basyskom-jvoe]

I have found the cause of the problem: My callback didn't set hasValue in the DataValue which caused a shallow copy of the DataValue being made in readValueAttributeFromDataSource() instead of the deep copy at https://github.com/open62541/open62541/blob/1.4/src/server/ua_services_attribute.c#L220

This is quite surprising to me because I remember that the UA_VARIANT_DATA_NODELETE mechanism used to be documented (at least in open62541 0.3 where I used it for the last time) as a way to do zero-copy reads. But if a deep copy is made directly after the data source read callback was called, it doesn't make any difference at all if I use UA_VARIANT_DATA_NODELETE or UA_Variant_setScalarCopy() in my read callback.

[jpfr]

_NODELETE can still be used for zero-copy reads.
Only MonitoredItems we persist the value so we are able to compare with the next sample.

But you make a good point.
We could check if the value is pointer-free (e.g. a number) and then do a memcpy over the already allocated previous memory...

[basyskom-jvoe]

@jpfr If I understand the current implementation correctly, a deep copy is always made directly after the data source read callback is called.
What you describe is how I remember the implementation in 0.3 (only the initial DataValue and any changed DataValue is deep copied, the comparison is done with the "original" without copying it first).

[jpfr]

Yeah. We had issues when keeping around _NODELETE variables after releasing the node in the nodestore.
So now we make a deep copy (if not already the case) internally.

What we can do is: Delay the node _release until after the MonitoredItem is fully processed.
For that we need some kind of global list (in the server struct) with to-be-released nodes.