Nodeset resolver Tool (for generating custom tailored namespace-zero nodesets) #4647

skuep2 · 2021-09-15T17:33:02Z

We are using the open62541 stack in an embedded environment, where RAM is scarce. When implementing a PA-DIM nodeset, I noticed that there are a lot of dependencies, that can thus far only be satisfied by including the full namespace-zero.
However, because this is a heavy burden on the on-chip memory I was looking for a way to produce a custom tailored namespace-zero that included only the hard dependencies.

I could not find a way to programatically produce such a custom XML file, which is why I created the nodeset_resolver tool. It heavily reuses the nodeset libraries used by the exquisite nodeset_compiler, which is why I put it into the same folder. Although this is a bit messy, this may be something a folder rename might fix in the future.

It works by loading the given XML files into a nodeset representation and then searches the tree for missing nodeIds. Then, the tool can look up those nodeIds from a reference XML, that is loaded into a separate nodeset representation. All (child-) dependencies are gathered and output. Lastly, it can automagically create an XML nodeset file for the required nodes from the reference XML or even splice it into the existing XML.

What do you think of this idea? It was a bit fiddly to get it working to really get all dependencies from the supplied nodeset files (especially if you don't know a lot about OPC UA yet) but the examples I have written into the README file work very well for our embedded setup. Comments are welcome!

EDIT: Should I include an affiliation in the README (similar to the one for the nodeset_compiler) in case you are interested in this PR?
EDIT2: Also a license should be included in the Python script, right?

lgtm-com · 2021-09-15T17:39:49Z

This pull request introduces 2 alerts when merging 91f28c9 into ee9d90d - view on LGTM.com

new alerts:

1 for Unused import
1 for Modification of parameter with default

lgtm-com · 2021-09-15T17:56:13Z

This pull request introduces 1 alert when merging 391ee1e into ee9d90d - view on LGTM.com

new alerts:

1 for Modification of parameter with default

GoetzGoerisch · 2021-09-15T18:33:57Z

Nice functionality!

But to my knowledge a compliant UA 1.04 server must provide all nodes defined by Part 3 -> NS0

Reference ConformanceUnit Address Space Base

This requirement is going away with UA 1.05. (See https://youtu.be/P-FQOhvBOBU?t=688)

skuep2 · 2021-09-15T18:44:39Z

Thanks for your reply!
I am in no way an expert on OPC UA, however if I understand correctly, the default reduced namespace that comes with open62541 passes the CTT (I assume that is the conformance test?).
See https://open62541.org/doc/1.2/building.html#build-options under UA_NAMESPACE_ZERO

The nodeset_resolver, when used with --merge just extends the reduced (or any for that matter) nodeset with the missing dependencies. I.e. the default reduced nodeset will always be a subset of the final (custom tailored) nodeset.
I guess this was not entirely clear in my first post.

jpfr · 2021-09-16T07:17:44Z

Wow. This looks amazing!

The examples uses only one -r reference file. Can several reference files be used as well?
From a developer perspective it might be easier to have a whitelist of NodeIds (e.g. a file with one NodeId per line) instead of a partial/invalid nodeset xml to be filled up. That should be at least on option. Note that we might have to use ExpandedNodeId because the namespace identifier is rewritten inside the nodeset compiler when several companion specs are loaded.
Yes, the license should be the MPv2 similar to the rest of the NodeSet compiler.

Looking forward to this!

… the nodeset_compiler

lgtm-com · 2021-09-16T11:48:16Z

This pull request introduces 1 alert when merging cb4e45c into ee9d90d - view on LGTM.com

new alerts:

1 for Modification of parameter with default

skuep2 · 2021-09-16T11:55:35Z

I am glad you find this tool useful! Thanks for your input.

* The examples uses only one `-r` reference file. Can several reference files be used as well?

In the beginning I had support for multiple reference files (to be loaded into the referenceNodeSet). However I ran into issues with the --pull functionality, where the missing nodes are pulled from the reference XML. This functionality would then be required to work with multiple reference files (i.e. looking for node Ids in multiple reference XML files), which introduces minor issues, e.g. what the behavior is when node Ids are found in multiple files. Anyway, this functionality is much more than we currently need, so I limited myself to just one reference XML file for now.

* From a developer perspective it might be easier to have a whitelist of NodeIds (e.g. a file with one NodeId per line) instead of a partial/invalid nodeset xml to be filled up. That should be at least on option.

I changed the output format of node ids (when not using --pull) to be one node id per line. I think that is what you mean with (generating?) a whitelist of NodeIds. (Or did you refer to reading NodeIds from a whitelist file?) E.g.:

./nodeset_resolver.py -e ../../tools/schema/Opc.Ua.NodeSet2.Reduced.xml \
    -x ../../deps/ua-nodeset/DI/Opc.Ua.Di.NodeSet2.xml \
    -x ../../deps/ua-nodeset/PADIM/Opc.Ua.IRDI.NodeSet2.xml \
    -x ../../deps/ua-nodeset/PADIM/Opc.Ua.PADIM.NodeSet2.xml \
    -r ../../deps/ua-nodeset/Schema/Opc.Ua.NodeSet2.xml
INFO:__main__:Preprocessing (existing) ../../tools/schema/Opc.Ua.NodeSet2.Reduced.xml
INFO:__main__:Preprocessing ../../deps/ua-nodeset/DI/Opc.Ua.Di.NodeSet2.xml
INFO:__main__:Preprocessing ../../deps/ua-nodeset/PADIM/Opc.Ua.IRDI.NodeSet2.xml
INFO:__main__:Preprocessing ../../deps/ua-nodeset/PADIM/Opc.Ua.PADIM.NodeSet2.xml
INFO:__main__:Collecting missing nodes...
INFO:__main__:Collected 28 missing nodes out of 619 used nodes
INFO:__main__:Preprocessing (reference) ../../deps/ua-nodeset/Schema/Opc.Ua.NodeSet2.xml
INFO:__main__:Resolving all dependencies from ../../deps/ua-nodeset/Schema/Opc.Ua.NodeSet2.xml...
INFO:__main__:Resolved 465 required nodes out of 1060 dependent nodes (0 unresolved)
ns=0;i=11616
ns=0;i=11715
ns=0;i=96
ns=0;i=95
ns=0;i=291
ns=0;i=256
ns=0;i=11508
ns=0;i=11510
ns=0;i=17603
ns=0;i=18347
ns=0;i=2771
ns=0;i=2307
ns=0;i=2311
ns=0;i=2310
ns=0;i=2309
[...]

Note that we might have to use ExpandedNodeId because the namespace identifier is rewritten inside the nodeset compiler when several companion specs are loaded.

I needed to think about that for a while.. You are totally right. So the reason the nodeset_resolver tool current works at all is, because I am only using nodeset-zero right? Or in other words, the output of the following command reports missing namespace Ids that do not correspond to actual namespace Ids in the OPC UA Specification (due to the rewriting you mention).

# removed -x Opc.Ua.Di.NodeSet2.xml to show issue
./nodeset_resolver.py -e ../../tools/schema/Opc.Ua.NodeSet2.Reduced.xml \
    -x ../../deps/ua-nodeset/PADIM/Opc.Ua.IRDI.NodeSet2.xml \
    -x ../../deps/ua-nodeset/PADIM/Opc.Ua.PADIM.NodeSet2.xml 
INFO:__main__:Preprocessing (existing) ../../tools/schema/Opc.Ua.NodeSet2.Reduced.xml
INFO:__main__:Preprocessing ../../deps/ua-nodeset/PADIM/Opc.Ua.IRDI.NodeSet2.xml
INFO:__main__:Preprocessing ../../deps/ua-nodeset/PADIM/Opc.Ua.PADIM.NodeSet2.xml
INFO:__main__:Collecting missing nodes...
INFO:__main__:Collected 20 missing nodes out of 251 used nodes
ns=0;i=11715
ns=0;i=11616
ns=0;i=256
ns=0;i=291
ns=0;i=17598
ns=0;i=17594
ns=0;i=17603
ns=2;i=15051
ns=0;i=17590
ns=0;i=17597
ns=2;i=15063
ns=2;i=6244
ns=2;i=1004
ns=0;i=17570
ns=0;i=19084
ns=0;i=15318
ns=0;i=2373
ns=0;i=19077
ns=0;i=2372
ns=0;i=11508

NodeIds from ns=2 are reported that actually belong to Opc.Ua.Di.NodeSet2.xml ~~that has ns=1 in the specification.~~
Do you have an example how ExpandedNodeId can be used to solve this problem?

EDIT: Had some more time, to think about it and I think I am mistaken in that there is an "official" namespace index in the specification. The namespace index of a node id used in XML files is only valid inside the XML file and references a (unique) namespace uri, as defined by the <NamespaceUris> ... </NamespaceUris> tags.

Thus, I assume with ExpandedNodeId that you refer to something like nsu=http:https://opcfoundation.org/UA/DI/;i=12345 to obtain a unique node id. Does it make sense to output this node id scheme? Can this scheme even be read by (e.g.) the nodeset_compiler (--blacklist)? From what I can see, nodeset.getNodeByIDString(...) supports Only (non-expanded) node ids (with the addition that ns=... can optionally also be a namespace uri.. Is this according to spec?)

Now that I think about it, the task of resolving dependencies is not easy, when the reference -r is NOT a namespace-zero. Currently, two separate NodeSet exist for the reference XML and the remaining XMLs. You would have to use ExpandedNodeIds (i.e. unique node ids) for resolving the dependencies between these nodesets. Maybe I will rename -r to --ns0 and claim that you can only use namespace-zero XMLs ;-) Unless there is someone with more OPC UA knowledge that wants to jump in here. As said, the tool suits our purpose of generating a new custom-tailored nodeset-zero XML.

skuep2 · 2021-09-16T15:29:44Z

# removed -x Opc.Ua.Di.NodeSet2.xml to show expanded output and added --expanded flag
./nodeset_resolver.py -e ../../tools/schema/Opc.Ua.NodeSet2.Reduced.xml \
    -x ../../deps/ua-nodeset/PADIM/Opc.Ua.IRDI.NodeSet2.xml \
    -x ../../deps/ua-nodeset/PADIM/Opc.Ua.PADIM.NodeSet2.xml --expanded
INFO:__main__:Preprocessing (existing) ../../tools/schema/Opc.Ua.NodeSet2.Reduced.xml
INFO:__main__:Preprocessing ../../deps/ua-nodeset/PADIM/Opc.Ua.IRDI.NodeSet2.xml
INFO:__main__:Preprocessing ../../deps/ua-nodeset/PADIM/Opc.Ua.PADIM.NodeSet2.xml
INFO:__main__:Collecting missing nodes...
INFO:__main__:Collected 20 missing nodes out of 251 used nodes
nsu=http:https://opcfoundation.org/UA/;i=11715
nsu=http:https://opcfoundation.org/UA/;i=11616
nsu=http:https://opcfoundation.org/UA/;i=256
nsu=http:https://opcfoundation.org/UA/;i=291
nsu=http:https://opcfoundation.org/UA/;i=17594
nsu=http:https://opcfoundation.org/UA/;i=17598
nsu=http:https://opcfoundation.org/UA/;i=17597
nsu=http:https://opcfoundation.org/UA/DI/;i=15063
nsu=http:https://opcfoundation.org/UA/;i=17590
nsu=http:https://opcfoundation.org/UA/;i=17603
nsu=http:https://opcfoundation.org/UA/DI/;i=15051
nsu=http:https://opcfoundation.org/UA/DI/;i=6244
nsu=http:https://opcfoundation.org/UA/DI/;i=1004
nsu=http:https://opcfoundation.org/UA/;i=17570
nsu=http:https://opcfoundation.org/UA/;i=19084
nsu=http:https://opcfoundation.org/UA/;i=15318
nsu=http:https://opcfoundation.org/UA/;i=2373
nsu=http:https://opcfoundation.org/UA/;i=19077
nsu=http:https://opcfoundation.org/UA/;i=2372
nsu=http:https://opcfoundation.org/UA/;i=11508

lgtm-com · 2021-09-16T15:35:33Z

This pull request introduces 1 alert when merging b74ccc8 into ee9d90d - view on LGTM.com

new alerts:

1 for Modification of parameter with default

…t node comparison (for namespaces other than namespace-zero)

…hout any arguments

lgtm-com · 2021-09-16T16:19:45Z

This pull request introduces 1 alert when merging a411a10 into ee9d90d - view on LGTM.com

new alerts:

1 for Modification of parameter with default

…l(...) function

…t may contain more namespace references than the original nodeset

lgtm-com · 2021-09-16T17:55:07Z

This pull request introduces 1 alert when merging 70229fc into 13212bd - view on LGTM.com

new alerts:

1 for Modification of parameter with default

lgtm-com · 2021-09-17T05:26:03Z

This pull request introduces 1 alert when merging 7858b49 into 13212bd - view on LGTM.com

new alerts:

1 for Modification of parameter with default

GoetzGoerisch · 2021-09-18T08:45:26Z

I am in no way an expert on OPC UA, however if I understand correctly, the default reduced namespace that comes with open62541 passes the CTT (I assume that is the conformance test?).
See https://open62541.org/doc/1.2/building.html#build-options under UA_NAMESPACE_ZERO

From the certified server:

open62541/CMakeLists.txt

Lines 152 to 154 in 1135945

 if(UA_ENABLE_MICRO_EMB_DEV_PROFILE) 

 if(NOT (UA_NAMESPACE_ZERO STREQUAL "FULL")) 

 message(FATAL_ERROR "CTT Compliant Micro Embedded Device Server Profile needs full namespace zero")

Therefore for UA 1.04 compliance a server product need to have a full NS0 for "Micro Embedded Server Profile" and above.

As mentioned earlier, this is a great feature for the upcoming UA 1.05 release, which is currently in review.

As Julius said, looking forward to this feature.

skuep2 · 2021-09-21T06:49:42Z

Thanks for the infos! I guess you are right, although the "problem" with non-conformity is not necessarily introduced by the nodeset_resolver, but by using the reduced namespace-zero in the first place (which is the default of open62541, if i remember correctly). The nodeset_resolver just extends the given namespace-zero by the dependencies.

PS: For someone with a heavy embedded background, it is kind of... interesting, that for something called a "micro embedded device profile", a full namespace-zero is required, comprising several megabytes of RAM. This is more RAM, than almost all non-application microcontrollers actually have on-chip.. by a multiple. :-)

nogryy · 2022-02-11T09:49:22Z

Hello,
very good feature!
With this it was for example possible to integrate the FileType support easily into the reduced namespace ns0.
Do you have any idea when this feature will be available in a release?
Greetings

skuep2 · 2022-07-07T13:32:49Z

Hello, very good feature! With this it was for example possible to integrate the FileType support easily into the reduced namespace ns0. Do you have any idea when this feature will be available in a release? Greetings

Sorry for the late reply, thanks for the kind words. We recently used the tool again for the designed purpose after an update and noticed a few problems where (apparently new) Aliases were not pulled into the final XML resulting in compile errors. I implemented the changes required for this problem.

Sorry about the force push :-X Looks like it destroyed the pull request history... EDIT: Restored

lgtm-com · 2022-07-07T13:36:15Z

This pull request introduces 1 alert when merging 7d020fe into 370b5f7 - view on LGTM.com

new alerts:

1 for Modification of parameter with default

lgtm-com · 2022-07-07T13:56:24Z

This pull request introduces 1 alert when merging 1177732 into c1f605a - view on LGTM.com

new alerts:

1 for Modification of parameter with default

jpfr · 2022-08-09T10:06:07Z

Hey there,

we had another look at this. This is really cool.

If we add a "whitelist" argument to the nodeset compiler, then we could use your walkNodes as a filter step to only select the nodes that are required for the whitelist.

If we move that internally to the nodeset compiler, that would forgeo the XML import-export intermediate step.
And we could load the full ns0 by default and remove everything that is not needed.

skuep2 · 2022-08-11T10:13:46Z

Hey Julius (former Fraunhofer Gesellschaft Colleague ;-)),
Thanks for your continued interest.
The tool was designed (as a separate tool) with the idea in mind, that I, as a developer using the open62541 library can call the nodeset_compiler.py on my own, to generate my own .c/.h nodeset files, that I will then integrate into my own application.

However, your (and other peoples) comments give me the impression that this is not the way open62541 was designed to operate. It looks like you are using the nodeset_compiler.py exclusively within the CMake environment, during the namespace-zero generation. And if you want to add other device models, they will be added to this namespace-zero generation (which technically, it is not a namespace-zero anymore in this case). For better understanding, is that what your comment is implying?
Anyway, I think the idea with a "whitelist" is good, is the term "whitelist" appropriate for what is happening behind the scenes?

Our use-case was, that we wanted to use the PADIM device information model, how would I find out, what nodes to whitelist to include anything to do with this particular information model? I think it would require a way of extracting all (?) nodes from the corresponding Opc.Ua.PADIM.NodeSet2.xml (and Opc.Ua.IRDI.NodeSet2.xml) to give to the compiler. Or would it be enough to just add the PADIM Object node id to the whitelist? For lack of deep insight into all the OPC UA stuff, I sincerely do not know if that would be enough. Or is there any obvious way I have missed?

Also another question I have is, do you suggest to implement walkNodes within nodeset_compiler.py to control the namespace-zero generation only or to generate a full nodeset, including PADIM+dependencies+stripped-ns0 as a single monolith integrated into the open26541 library/amalgamate?

I have to admit, I am not that deeply invested into the OPC UA/open62541 development and I am not familiar with certain ways to do or ways, that things were designed to be done (Which is why I never thought about integrating this into the nodeset_compiler). Please excuse my ignorance in some of the points above.

Feel free to use the code! If you need me to do something (sourcecode or organizational stuff) let me know.

jpfr · 2022-08-12T14:57:21Z

Hi @skuep2 ,

the node set compiler can also be used standalone, not only within cmake.
The question is whether your tool is to be used outside of the open62541 ecosystem as well.
Then it makes sense to have XML as an exchange format.

If you just write out the xml to read it back into the node set compiler, then we can reduce the complexity and omit the intermediate xml representation.

The whitelisiting could be done for a list of nodes. Or, since every companion spec has its own namespace (uri), on the level of namespaces.

thanks for giving the go-ahead for working with your code. I think this will become a new feature rather soon.

skuep2 · 2022-08-15T14:40:39Z

I think it would be perfectly reasonable to bind the tool to the open62541 ecosystem. The reason I chose XML as the exchange format was, that there was/is no other way to use a customized nodeset-zero than giving my-custom-ns0.xml to cmake (besides from choosing one of the pre-made ones).

But yeah, during cmake/build this XML will be read again by the nodeset compiler, you are totally right.
I guess you would want to add some compile options to cmake so you can a) enable tailored-nodeset-zero generation and b) control what nodes are allowed, right?
I like the idea with the namespace (uri)! I did not think of it that way.

Best Regards

skuep2 added 11 commits September 15, 2021 10:24

Initial working commit

be16f40

Refactored variables and added comments

3b2cdea

Improved NodeId XML filtering

3fa24bb

Added argument for controlling output

6753e98

Restructured XML printing

5c5605f

Changed wording of log output

1785401

Implemented proper XML filtering/merging by resorting to lxml

ce59001

Added documentation on nodeset_resolver usage

7f72511

Adjusted help message text

2388de9

Adjusted README.md to updated help text

fe15310

Added more information on how to use the generated XML files

91f28c9

Fixed code alerts and removed remnant comments

391ee1e

skuep2 added 3 commits September 16, 2021 13:23

Added file header and MPLv2 license

0ebc06f

Added affiliation for nodeset_resolver

82f4728

Node ids are now output as one node id per line to be compatible with…

cb4e45c

… the nodeset_compiler

skuep2 added 3 commits September 16, 2021 17:23

Included -u flag to output expanded node ids instead of regular node ids

4fba77c

Updated help output

1d4a4b7

Updated README with new -u flag

b74ccc8

skuep2 added 3 commits September 16, 2021 17:52

Set namespace of reference nodeset to existing nodeset to allow direc…

af74036

…t node comparison (for namespaces other than namespace-zero)

Added capability for multiple reference files

a1ac002

Updated readme with new help output

992e55b

Removed unncessary .format(...) resulting in errors when launched wit…

a411a10

…hout any arguments

skuep2 added 4 commits September 16, 2021 19:37

Introduced rudimentary support for multiple reference XMLs in printXm…

3a58f12

…l(...) function

Fixed bad logging output

d499ee7

Improved error message output when no input files could be loaded

dae713f

Use referenceNodeSet namespace for looking up ExpandedNodeIds since i…

70229fc

…t may contain more namespace references than the original nodeset

Do not use referenceNodeSet when -r is not specified on command line

7858b49

andreasebner assigned NoelGraf May 10, 2022

skuep2 force-pushed the nodeset-resolver branch from 9e01679 to 7d020fe Compare July 7, 2022 13:29

Implemented pulling/merging of <Models> and <Aliases>

1177732

skuep2 force-pushed the nodeset-resolver branch from 7d020fe to 1177732 Compare July 7, 2022 13:47

skuep2 mentioned this pull request Jul 11, 2022

Using a custom (reduced) NS0 but include all UA_TYPES during cmake (for low-memory architectures) #5252

Open

skuep2 closed this Jul 21, 2022

skuep2 deleted the nodeset-resolver branch July 21, 2022 10:47

skuep2 restored the nodeset-resolver branch July 21, 2022 10:55

skuep2 reopened this Jul 21, 2022

andreasebner added the Component: Nodeset Compiler Issues related to the nodeset compiler label Aug 15, 2022

jpfr mentioned this pull request Jan 30, 2023

Ongoing Work, Feature Wishlist and Prioritization #5075

Open

33 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Nodeset resolver Tool (for generating custom tailored namespace-zero nodesets) #4647

Nodeset resolver Tool (for generating custom tailored namespace-zero nodesets) #4647

skuep2 commented Sep 15, 2021 •

edited

Loading

lgtm-com bot commented Sep 15, 2021

lgtm-com bot commented Sep 15, 2021

GoetzGoerisch commented Sep 15, 2021

skuep2 commented Sep 15, 2021 •

edited

Loading

jpfr commented Sep 16, 2021

lgtm-com bot commented Sep 16, 2021

skuep2 commented Sep 16, 2021 •

edited

Loading

skuep2 commented Sep 16, 2021 •

edited

Loading

lgtm-com bot commented Sep 16, 2021

lgtm-com bot commented Sep 16, 2021

lgtm-com bot commented Sep 16, 2021

lgtm-com bot commented Sep 17, 2021

GoetzGoerisch commented Sep 18, 2021

skuep2 commented Sep 21, 2021 •

edited

Loading

nogryy commented Feb 11, 2022

skuep2 commented Jul 7, 2022 •

edited

Loading

lgtm-com bot commented Jul 7, 2022

lgtm-com bot commented Jul 7, 2022

jpfr commented Aug 9, 2022

skuep2 commented Aug 11, 2022

jpfr commented Aug 12, 2022

skuep2 commented Aug 15, 2022

Nodeset resolver Tool (for generating custom tailored namespace-zero nodesets) #4647

Are you sure you want to change the base?

Nodeset resolver Tool (for generating custom tailored namespace-zero nodesets) #4647

Conversation

skuep2 commented Sep 15, 2021 • edited Loading

lgtm-com bot commented Sep 15, 2021

lgtm-com bot commented Sep 15, 2021

GoetzGoerisch commented Sep 15, 2021

skuep2 commented Sep 15, 2021 • edited Loading

jpfr commented Sep 16, 2021

lgtm-com bot commented Sep 16, 2021

skuep2 commented Sep 16, 2021 • edited Loading

skuep2 commented Sep 16, 2021 • edited Loading

lgtm-com bot commented Sep 16, 2021

lgtm-com bot commented Sep 16, 2021

lgtm-com bot commented Sep 16, 2021

lgtm-com bot commented Sep 17, 2021

GoetzGoerisch commented Sep 18, 2021

skuep2 commented Sep 21, 2021 • edited Loading

nogryy commented Feb 11, 2022

skuep2 commented Jul 7, 2022 • edited Loading

lgtm-com bot commented Jul 7, 2022

lgtm-com bot commented Jul 7, 2022

jpfr commented Aug 9, 2022

skuep2 commented Aug 11, 2022

jpfr commented Aug 12, 2022

skuep2 commented Aug 15, 2022

skuep2 commented Sep 15, 2021 •

edited

Loading

skuep2 commented Sep 15, 2021 •

edited

Loading

skuep2 commented Sep 16, 2021 •

edited

Loading

skuep2 commented Sep 16, 2021 •

edited

Loading

skuep2 commented Sep 21, 2021 •

edited

Loading

skuep2 commented Jul 7, 2022 •

edited

Loading