[WIP] feat(pubsub): Publishing of Events #4220

florianfischeer · 2021-03-03T12:44:45Z

This pull request adds the publishing of events to the PubSub part of the stack.
We added a cmake-flag, which is called UA_ENABLE_PUBSUB_EVENTS, a tutorial for publishing and another one for subscribing.

Concept

In the PubSubManager we added a list that saves all PublishedDataSets of the type UA_PUBSUB_DATASET_PUBLISHEDEVENTS with their associated DataSetWriters. Besides, a DataSetWriter now contains a queue in which the information of the events is saved and then sent after the publishing interval has expired. When creating a DataSetWriter, if the associated PublishedDataSet is of the mentioned type, both are added to the PubSubManager list. This list helps us to recognize when an event has occurred whether it belongs to a PublishedDataSet. If so, the DataSetFields are added to the queue of the DataSetWriter belonging to the PublishedDataSet.

Specific Changes

Added structs

To implement our concept, we had to create some new structs. First, we needed a UA_DataSetEventConfig, because there was only a placeholder in the UA_DataSetFieldConfig. From our point of view, an EventDataSetField contains a selectedField, a fieldname and a boolean, which shows, whether it's a promoted field. Also, we added a pointer of selectedFields to the UA_PublishedEventConfig.

Moreover, we needed some structs to implement the Queue in the DataSetWriter and the list in the PubSubManager. The Queue in the DataSetWriter consists of EventQueueEntries, which are only a wrapper for DataValues. The list of the PubSubManager on the other hand consists of PublishedDataSetEventEntries. This struct contains a field for a DataSetWriter and one for its connected PublishedDataSet.

Changed Methods

For our implementation, the most important method is UA_Server_triggerEvent, because it's necessary to call it if you want to trigger an event. In this method, we check, whether the triggered Event matches with a PublishedDataSet and if so we call addEventToDataSetWriter. This method is new and it iterates over all DataSetFields of the given PublishedDataSet, wraps them into a DataValue and passes it to the insertDataValueIntoDSWQueue-method. In here the passed DataValue is added to the Queue in the passed DataSetWriter.

If the publishing interval expired and a KeyFrameMessage needs to be created, we fill the DataSetMessage with the DataValues in the Queue. After this, the Queue is cleared and all information is published.

Additions to the Informationmodel

We added functionality to support PublishedDataSets of the PublishedEventsType in the information model. Most of the functionality is implemented analogue to the PublishedDataItemsType.

Added Methods

The first method we added is addPublishedEventsRepresentation which is called every time a PDS is created by UA_Server_addPublishedDataSet, provided that the information model is enabled and the created PDS is of the PublishedEventsType. This method adds the required nodes to represent a PublishedEventsType Object to the information model.

We also implemented the addPublishedEvents method for the DataSetFolderType (9.1.4.5.3) which, when called, creates a PDS of PublishedEventsType.

Added to the NodeSet files

To fully support the PublishedEventsType even when running at reduced ns0 we had to add a few nodes to the NodeSet files. More precisely we added AddPublishedEvents and PublishedEventsType to Opc.Ua.NodeSet2.PubSubMinimal.xml and dependent nodes to Opc.Ua.NodeSet2.Reduced.xml.

There are a few points that we mention here that are currently in progress and not merged into this branch

CLAassistant · 2021-03-03T12:44:49Z

All committers have signed the CLA.

jpfr · 2021-03-03T21:14:18Z

Very good feature and also code quality from a first look.

@andreasebner can we green-light the public API before doing an in-depth review?

florianfischeer · 2021-03-04T19:10:14Z

Thanks for your comment on our pull request. After a meeting with @andreasebner we recognized, that there still need to be made some adjustments to match the feature with the specification.

andreasebner · 2021-03-07T22:05:42Z

Very good feature and also code quality from a first look.

@andreasebner can we green-light the public API before doing an in-depth review?

Yes, for sure, I will trigger you if we reach the final stage. The public API should not be changed a lot since events in PubSub are "only" another type of PublishedDataSets.

andreasebner · 2021-03-07T22:08:21Z

@Florianfischer42 Please provide a description for this PR. The description should explain your work, refer to the relevant PubSub specification sections and list changes/additions on the public API.

jpfr · 2021-03-30T20:28:12Z

Looking good! Two quick questions:

Is the eventQueue protected against overflow? That is, if events are generated faster than we can send them out, where will things break?
You manage a global list of PublishedDataSetEventEntry.
- Iterating over the global list for all events could become inefficient when the number of PublishedDataSetEventEntry gets large.
- Does that cover Events that "blubble up" in the object hierarchy?
  You might want to reuse functionality from the Event-MonitoredItems for that.
  They are added to a linked-list of the node where they "collect" events bubbling up from below objects in the hierarchy.

Keep up the good work!

florianfischeer · 2021-04-02T13:11:23Z

@jpfr
Thank you for the feedback.

Is the eventQueue protected against overflow? That is if events are generated faster than we can send them out, where will things break?

We added a basic overflow protection, which means if the queue is full or memory could not be allocated it will just fail to add more events and return an error code.

Iterating over the global list could become inefficient

For now, a global list that contains event-only PDS is the most efficient way we could come up with. If you can suggest a better solution, please let us know.

Are events bubbling up in the object hierarchy?

That's a good point, bubbling up is now supported.

andreasebner

Thanks’ for your great contribution!

I've added some open questions and mentioned some problems/formatting issues. In addition please clean up your commit history.

andreasebner · 2021-04-04T21:56:10Z

examples/pubsub/tutorial_pubsub_publish_event.c

+ * Copyright (c) 2021 Stefan Joachim Hahn, Technische Hochschule Mittelhessen
+ * Copyright (c) 2021 Florian Fischer, Technische Hochschule Mittelhessen


Examples (CC licence) do not mention the autor (compare with other examples). This comment applys also to other files.

andreasebner · 2021-04-04T22:03:04Z

src/server/ua_subscription_events.c

+ * Copyright (c) 2021 Stefan Joachim Hahn, Technische Hochschule Mittelhessen
+ * Copyright (c) 2021 Florian Fischer, Technische Hochschule Mittelhessen


Please unify the includes.

andreasebner · 2021-04-05T20:25:33Z

examples/pubsub/tutorial_pubsub_publish_event.c

+ selectedFields[0].typeDefinitionId = UA_NODEID_NUMERIC(0, UA_NS0ID_BASEEVENTTYPE);
+ selectedFields[0].attributeId = UA_ATTRIBUTEID_VALUE;
+ selectedFields[0].browsePathSize = 1;
+ UA_QualifiedName browsePathMessage[1];
+ browsePathMessage[0] = UA_QUALIFIEDNAME(0, "Message");
+ selectedFields[0].browsePath = browsePathMessage;
+
+ selectedFields[1].typeDefinitionId = UA_NODEID_NUMERIC(0, UA_NS0ID_BASEEVENTTYPE);
+ selectedFields[1].attributeId = UA_ATTRIBUTEID_VALUE;
+ selectedFields[1].browsePathSize = 1;
+ UA_QualifiedName browsePathSeverity[1];
+ browsePathSeverity[0] = UA_QUALIFIEDNAME(0, "Severity");
+ selectedFields[1].browsePath = browsePathSeverity;
+


You can move the repeating things to the for loop.

andreasebner · 2021-04-05T20:31:13Z

examples/pubsub/tutorial_pubsub_publish_event.c

+}
+
+/*********************** Event Methods ***********************/
+void callEvent(UA_Server *server, void *data);


Avoid forward declaration here and move the function to this location.

andreasebner · 2021-04-05T20:33:02Z

examples/pubsub/tutorial_pubsub_publish_event.c

+ addDataSetFields(server);
+ addWriterGroup(server);
+ addDataSetWriter(server);
+ const UA_Double interval = (UA_Double)5000;


Please declare a define for this config parameter.

andreasebner · 2021-04-05T20:56:53Z

src/pubsub/ua_pubsub_manager.h

@@ -16,12 +16,24 @@ _UA_BEGIN_DECLS

 #ifdef UA_ENABLE_PUBSUB /* conditional compilation */

+#ifdef UA_ENABLE_PUBSUB_EVENTS
+typedef struct PublishedDataSetEventEntry {
+ UA_PublishedDataSet *pds;


Why is the pds needed? The dws always reffer to a pds.

It isn't needed. We added it, to improve runtime. The PublishedDataSet can't be accessed directly from a DSW, it only offers the NodeId of a PDS. The algorithm to find the PDS (UA_PublishedDataSet_findPDSbyId) iterates over all PDSs, this would affect the efficiency.

andreasebner · 2021-04-05T21:02:40Z

src/pubsub/ua_pubsub_ns0.c

+ retVal |= UA_Server_addReference(server, publishedDataSet->identifier,
+ UA_NODEID_NUMERIC(0, UA_NS0ID_HASCOMPONENT),
+ UA_EXPANDEDNODEID_NUMERIC(0, UA_NS0ID_PUBLISHEDEVENTSTYPE_MODIFYFIELDSELECTION), true);


I think you currently did not add the implementation of the 'MODIFYFIELDSELECTION' function, right?

Right, it's still on the todo list. I commented it out with a todo notice. Probably makes sense to change the event config from DSF to PDS before implementing this method.

andreasebner · 2021-04-05T21:09:48Z

src/pubsub/ua_pubsub_writer.c

+ } else if(newField->config.dataSetFieldType == UA_PUBSUB_DATASETFIELD_EVENT){
+ if(newField->config.field.events.promotedField)
+ currentDataSet->promotedFieldsCount++;
+ currentDataSet->fieldSize++;


Is it necessary to call addDataSetField after creating an Event-PDS?

Currently yes, it sets the correct PDS fieldSize, the promotedFields and generates the FieldMetaData. It wouldn't be too difficult to rewrite it to PDS level though.

andreasebner · 2021-04-05T21:10:39Z

src/pubsub/ua_pubsub_writer.c

+#ifdef UA_ENABLE_PUBSUB_EVENTS
+ if(currentDataSetContext->config.publishedDataSetType == UA_PUBSUB_DATASET_PUBLISHEDEVENTS){
+ SIMPLEQ_INIT(&newDataSetWriter->eventQueue); // only needs to be initalized if the linked publishedDataSet is for event
+ newDataSetWriter->eventQueueMaxSize = 1000; //test value


This parameter should be more visible.

andreasebner · 2021-04-05T21:14:02Z

src/server/ua_subscription_events.c

+ UA_LOG_ERROR(&server->config.logger, UA_LOGCATEGORY_SERVER,
+ "EventQueueMaxSize reached %lu / %lu", dsw->eventQueueEntries, dsw->eventQueueMaxSize);


I think a warning is enough. Just drop the new field or remove the oldest entry and add this new one.

- bubbling-up - moves eventQueueMaxSize to config - removes reference to unimplemented ModifyFieldSelection - adds comments for the global event-PDS list

AlexexanderSavinov · 2021-06-07T11:01:28Z

This one is still open as i understand.
I am currently on master branch and use PubSub and need this changes.
Is there a plan when they are going to be included?

CodingLappen · 2021-06-23T09:06:37Z

I'd also like to know when this commit will be merged or if it will be merged.

jpfr · 2021-10-05T06:21:35Z

@Florianfischer42 Are you still on this?
There are a few review comments from @andreasebner still unaddressed.

applea9 · 2021-11-22T10:05:44Z

I am also interested in this PR, is there any plan to merge this PR to master?

wlkrm · 2022-04-05T08:13:26Z

I interested in this PR, too. Is there any plan to merge this PR to master? If you have
no plans in continuing this PR, I might address the unaddressed review comments, @florianfischeer ?

jpfr · 2022-04-12T15:14:48Z

As @florianfischeer is not responsing, I think you can take this up @wlkrm !

florianfischeer force-pushed the cleanup branch from a078bd0 to a4dcd52 Compare March 5, 2021 10:50

andreasebner mentioned this pull request Mar 7, 2021

[WIP] feat(pubsub): Publish events #4192

Closed

florianfischeer force-pushed the cleanup branch 5 times, most recently from bc939f7 to f42124d Compare March 18, 2021 16:38

florianfischeer force-pushed the cleanup branch 3 times, most recently from b23ac69 to 86bc5dc Compare March 28, 2021 18:03

andreasebner reviewed Apr 5, 2021

View reviewed changes

florianfischeer and others added 7 commits April 8, 2021 15:23

feat(pubsub): Publish Events

899d922

feat(pubsub): Events Informationmodel-Methods

76518b6

test(pubsub): Publish Events

fd642e7

feat(pubsub): FieldMetaData for Event-DSF

2d4f635

feat(pubsub): Evaluation of Event Contentfilter

56f277f

feat(pubsub): Events in config-files

6a55fea

changes from code review

1b36d3b

- bubbling-up - moves eventQueueMaxSize to config - removes reference to unimplemented ModifyFieldSelection - adds comments for the global event-PDS list

florianfischeer force-pushed the cleanup branch from 8ffa79d to 1b36d3b Compare April 8, 2021 14:24

fixes problem with updated transport profile

452a7bb

jpfr mentioned this pull request Apr 22, 2022

Ongoing Work, Feature Wishlist and Prioritization #5075

Open

33 tasks

andreasebner modified the milestone: 1.4 Apr 27, 2022

andreasebner self-assigned this Apr 27, 2022

andreasebner added Component: PubSub labels Apr 27, 2022

jpfr added the Status: Help wanted Looking for support from the community to implement/fix this issue. Also tag with 'help wanted' label Jan 30, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[WIP] feat(pubsub): Publishing of Events #4220

[WIP] feat(pubsub): Publishing of Events #4220

florianfischeer commented Mar 3, 2021 •

edited

Loading

CLAassistant commented Mar 3, 2021 •

edited

Loading

jpfr commented Mar 3, 2021

florianfischeer commented Mar 4, 2021

andreasebner commented Mar 7, 2021

andreasebner commented Mar 7, 2021

jpfr commented Mar 30, 2021

florianfischeer commented Apr 2, 2021

andreasebner left a comment •

edited

Loading

andreasebner Apr 4, 2021

andreasebner Apr 4, 2021

andreasebner Apr 5, 2021

andreasebner Apr 5, 2021

andreasebner Apr 5, 2021

andreasebner Apr 5, 2021

florianfischeer Apr 7, 2021

andreasebner Apr 5, 2021

Tym0r Apr 12, 2021

andreasebner Apr 5, 2021

Tym0r Apr 7, 2021 •

edited

Loading

andreasebner Apr 5, 2021

andreasebner Apr 5, 2021

AlexexanderSavinov commented Jun 7, 2021

CodingLappen commented Jun 23, 2021 •

edited

Loading

jpfr commented Oct 5, 2021

applea9 commented Nov 22, 2021

wlkrm commented Apr 5, 2022

jpfr commented Apr 12, 2022

		* Copyright (c) 2021 Stefan Joachim Hahn, Technische Hochschule Mittelhessen
		* Copyright (c) 2021 Florian Fischer, Technische Hochschule Mittelhessen

		UA_LOG_ERROR(&server->config.logger, UA_LOGCATEGORY_SERVER,
		"EventQueueMaxSize reached %lu / %lu", dsw->eventQueueEntries, dsw->eventQueueMaxSize);

[WIP] feat(pubsub): Publishing of Events #4220

Are you sure you want to change the base?

[WIP] feat(pubsub): Publishing of Events #4220

Conversation

florianfischeer commented Mar 3, 2021 • edited Loading

Concept

Specific Changes

Added structs

Changed Methods

Additions to the Informationmodel

Added Methods

Added to the NodeSet files

CLAassistant commented Mar 3, 2021 • edited Loading

jpfr commented Mar 3, 2021

florianfischeer commented Mar 4, 2021

andreasebner commented Mar 7, 2021

andreasebner commented Mar 7, 2021

jpfr commented Mar 30, 2021

florianfischeer commented Apr 2, 2021

Is the eventQueue protected against overflow? That is if events are generated faster than we can send them out, where will things break?

Iterating over the global list could become inefficient

Are events bubbling up in the object hierarchy?

andreasebner left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Tym0r Apr 7, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

AlexexanderSavinov commented Jun 7, 2021

CodingLappen commented Jun 23, 2021 • edited Loading

jpfr commented Oct 5, 2021

applea9 commented Nov 22, 2021

wlkrm commented Apr 5, 2022

jpfr commented Apr 12, 2022

florianfischeer commented Mar 3, 2021 •

edited

Loading

CLAassistant commented Mar 3, 2021 •

edited

Loading

andreasebner left a comment •

edited

Loading

Tym0r Apr 7, 2021 •

edited

Loading

CodingLappen commented Jun 23, 2021 •

edited

Loading