Historize data in a circular buffer

[luibass92]

Goodmorning,

I am a "Historical Data" plugin user and I see that there is no limit on the size of the buffer which historizes data.
I can set an initial size but, each time it is reached, a UA_realloc is done and the buffer is re-allocated with its size doubled.

I would like to configure the historical data in order to have something similar to a circular buffer beacause I want to be sure to not have undesired memory growths.

I managed how to do it with the Gathering, now I can register only a maximum number of nodes and if I overcome that limit an UA_STATUS_BADOUTOFMEMORY is returned.
Now I am trying to do the same with the Backend part but it seems way more complicated.
In this case I do not want to simply return a bad status code, but I want to historize data in a circular buffer which does not double its size but overwrite the oldes data as a circlar buffer would do.

Someone could please help me or give me some tips?

Thanks,
Luigi

[jpfr]

@peter-rustler-basyskom
@basyskom-meerkoetter

Can you have a look at this?

[gcn1996]

I have the same question,it will take too much memory and break the program

[luibass92]

Hi everyone,
after several months I have implemented by myself a solution to have a "circular history".

The idea behind it is to have a fixed number of nodes to historize, that means it is mandatory to choose a maximum number of nodes (and that number will never be exceeded) and for each of that nodes a circular array of values will be saved and, each time a client performs a UA_HistoryReadRequest on a certain node, the current circular buffer (full of values) will be sent by the server.

To do so I started from the original history functions and I did some modifications to obtain the desired result. This was the faster solution for me but it means that most of the code it is very similar to the original implementation. Sorry for that but I did not find a faster way to implement the algorithms.

The following images describe the differences between the original history and the "circular" one.

Original implementation of OPC UA Historical Data Access: nodes and
corresponding values can grow indefinitely:

Modified implementation of OPC UA Historical Data Access: if max
number of nodes is reached (in the image is 3), registration of other nodes is not allowed. If max
number of values is reached (in the image is 4), subsequent values are saved overwriting oldest ones
following the rules of circular buffers

Moreover it is useful to know that I have added a field to the UA_NodeIdStoreContextItem_backend_memory struct in order to make things easier to implement.

To try this implementation all you need to do is to set:

serverSetHistoryData = &serverSetHistoryData_backend_memory_Circular;
getHistoryData = &getHistoryData_service_Circular;
registerNodeId = &registerNodeId_gathering_Circular;

While the History Database part (interface) will remain untouched.

Find attached a zip file with the code, your opinions would be very appreciated ^_^

circular_history_open62541.zip

@jpfr
@Pro
@peter-rustler-basyskom
@basyskom-meerkoetter

[Max15421]

Hi I am trying to implement a circular buffer. But I cannot find the functions UA_DataValueMemoryStoreItem_deleteMembers and UA_NodeIdStoreContextItem_deleteMembers. Did you write them yourself? They are not in the ZIP file. Or is UA_DataValueMemoryStoreItem_deleteMembers =UA_DataValueMemoryStoreItem_clear and UA_NodeIdStoreContextItem_deleteMembers =UA_NodeIdStoreContextItem_clear?
Thanks a lot!

[luibass92]

Hi I am trying to implement a circular buffer. But I cannot find the functions UA_DataValueMemoryStoreItem_deleteMembers and UA_NodeIdStoreContextItem_deleteMembers. Did you write them yourself? They are not in the ZIP file. Or is UA_DataValueMemoryStoreItem_deleteMembers =UA_DataValueMemoryStoreItem_clear and UA_NodeIdStoreContextItem_deleteMembers =UA_NodeIdStoreContextItem_clear? Thanks a lot!

Hi @Max15421,
first of all thanks for re-opening this topic, it seems that not so many people need to avoid memory leaks in their code 😄
As far as I remember I have been using the default functions UA_DataValueMemoryStoreItem_deleteMembers and UA_NodeIdStoreContextItem_deleteMembers .
This is a general rule, if I did not implement some of the functions it means that or I am not calling them at all (in the interface they are set to NULL), or I am using the default ones.

Have you managed to compile the newest version of open62541 with these modifications?

[Max15421]

@luibass92 Thanks for your answer. I was able to compile it after some changes. I exchanged UA_DataValueMemoryStoreItem_deleteMembers and UA_NodeIdStoreContextItem_deleteMembers with UA_DataValueMemoryStoreItem_clear and UA_NodeIdStoreContextItem_clear because the functions were depricated. I also changed OPCUA_NULL to NULL. Unfortunately it doesn't work. I get an memory overflow after a while. I wasn't able to find the mistake yet.

[luibass92]

@Max15421 yes, I implemented the circular buffer several open62541 versions ago...
But I guess the problem should be in serverSetHistoryData_backend_memory_Circular.

In fact if you take a look at the final lines of this method, you should see:

if (item->dataStore[item->lastInserted] != OPCUA_NULL)
{
	UA_DataValueMemoryStoreItem_deleteMembers(item->dataStore[item->lastInserted]);
	UA_free(item->dataStore[item->lastInserted]);
}
item->dataStore[item->lastInserted] = newItem;
++item->lastInserted;

And the use of UA_DataValueMemoryStoreItem_deleteMembers is crucial at this point. It is needed to replace an old value in the circular buffer with the new one.

Does not exist a new method that make the same job of UA_DataValueMemoryStoreItem_deleteMembers ?

Let me know if this help.
Thanks.

[Max15421]

@luibass92 thanks!
UA_DataValueMemoryStoreItem_clear makes the job of UA_DataValueMemoryStoreItem_deleteMembers. I don't know what went wrong when I implemented it the first time but now it works fine. To sum up for the current version you have to change:

1. UA_DataValueMemoryStoreItem_deleteMembers to UA_DataValueMemoryStoreItem_clear
2. UA_NodeIdStoreContextItem_deleteMembers to UA_NodeIdStoreContextItem_clear
3. UA_ByteString_deleteMembers to UA_ByteString_clear
4. OPCUA_NULL to NULL

[luibass92]

I finally decided to create a pull request after adapting the code with the latest version of the library.
You can find the pull request here.
Thanks for your support :)

[luibass92]

Merged on master #4740.

Closing this issue.