About "opendssdirect fails at this"

[PMeira]

Hi, I found this comment through GitHub's search:

# Execute opendss's save command reliably on a circuit. opendssdirect fails at this.

As one of the maintainers of OpenDSSDirect.py/DSS-Python, does this happen with recent versions? If so, could you share a circuit that fails so we could investigate and fix it?

By the way, we have a newish JSON export implementation that should work for most components (doesn't include buses or general metadata yet, but should be easier to get those through the API): https://github.com/dss-extensions/dss_python/blob/master/docs/examples/JSON.ipynb

#TODO: Fix repeated wdg= keys!?!?!?

For Transformer/XfmrCode wdg, wdg is used to select the winding, similar to cond in LineGeometry. In the JSON export, since yesterday's releases of DSS C-API and DSS-Python, those are hidden and the winding/conductor data is returned as arrays. We plan to provide some more import/export alternatives in the future.

[dpinney]

Hello! First off, thank you for all your work on opendssdirect, it's a wonderful tool. And also thank you for your great online opendss tutorials.

Our app's users need the ability to manipulate opendss circuits in our browser-based GIS editor, so we had to write a decent .dss <--> json bidirectional converter. Our first attempt was done by reading opendssdirect's in-memory representation of the .dss files, but we saw a lot of instances where powerflow would fail when we went dss --> json --> dss, so we wrote this new (super hacky) converter that has been more reliable for us. These are the test files we have been using.

It sounds like we should give opendssdirect another shot here. Although we have also had issues where opendsscmd would solve circuits where opendssdirect failed, so we've mostly tried to standardize on raw opendss to limit the amount of tooling we need to debug through.

The repeated wgd syntax... since we want to represent each opendss object as a dict/object/hashmap we don't support repeated keys in our json representation and prefer the array syntax. We would have to upgrade our parser to support translating repeated keys to array syntax, which is currently on the back burner.

Our ideal solution would be support for json representations in opendss itself, or at least some kind of 'use strict' export functionality that would get arbitrary .dss file collections into a format that parses easily, reliably and consistently. I can dream right? (No disrespect intended towards opendss here. We've been writing equally egregious hacks for gridlabd and matpower formats as well.)

[PMeira]

Thanks for the links! I'll try to test soon to see what I can find. Nowadays, for example, I know that our engine have success saving some circuits where the official version fails.

And also thank you for your great online opendss tutorials.

You're probably thinking of the other Paulo who works at EPRI. I did post a lot in the OpenDSS discussion forum until last year. Sorry for the wall of text below, I should probably preface with a bit of explanation, there's also a FAQ in https://github.com/dss-extensions/dss-extensions

• In Feb 2018, we announced an alternative C API based on the work we'd been doing in Unicamp/Brazil since around 2016 to workaround issues found in the official OpenDSSDirect.DLL. That's the DSS C-API library. We posted in the forum, but even after some adoption, the patches weren't accepted for the official version.
• A few months later, still in 2018, OpenDSSDirect.py ("ODD.py" for short) was migrated to use DSS C-API. Some time later so was OpenDSSDirect.jl (the names are a bit unfortunate). A lot of NREL projects are based on ODD.py.
• Since then, we've been slowly adding more error-checking, validation, optimizations, and features to this alternative implementation, besides refactoring the code. We did report many issues in the past, but there's no public issue/bug tracker for OpenDSS, so some are lost/forgotten. The current goal is to finish refactoring the rest of the code to finish a C++ port, besides documenting the extra features.

AFAIK, there are some variations of OpenDSS in the wild:

• The official OpenDSS provided by EPRI, Windows-only, in forms of EXE, COM DLL, and "somewhat" native DLL (besides some other variations used internally by EPRI). This is the only one supported by EPRI. For users that need EPRI support, we usually suggest either a dedicated Windows VM, or running through Wine.
• Our projects from DSS-Extensions (DSS C-API, DSS-Python, OpenDSSDirect.py, OpenDSSDirect.jl, DSS Sharp, DSS MATLAB, dss.hpp).
• OpenDSSCmd and variations
• MA-OpenDSS (developed at UCF, doesn't look like an active project anymore)

I imagine the official version and the alternative from DSS-Extensions are the two most widely used and tested. OpenDSSCmd, although built with Free Pascal on Linux/macOS, does not share any code with DSS C-API, which includes a lot of bug fixes for Linux, macOS and FPC.

Since the code and features have grown, we'll probably adopt the name "AltDSS" soon to make it clearer for users and address concerns communicated by EPRI, but I'm not sure we can/should change the names for OpenDSSDirect.py and OpenDSSDirect.jl (I'll have to discuss with the other devs).

Although we have also had issues where opendsscmd would solve circuits where opendssdirect failed

Maybe it's a bug we didn't catch, it could have been a bug we already fixed, or it could be that OpenDSSCmd was using an outdated version of OpenDSS (it also just ignores some errors silently sometimes). The engine from DSS-Extensions is cross-validated with nearly all sample circuits from OpenDSS + some private data. There's a lot of bugs specific to Free Pascal or Linux/macOS, or even ARM vs x86 issues. EPRI doesn't support those and, since there are usually more important bugs, we fix them in our version but usually don't report.

The repeated wgd syntax... since we want to represent each opendss object as a dict/object/hashmap we don't support repeated keys in our json representation and prefer the array syntax. We would have to upgrade our parser to support translating repeated keys to array syntax, which is currently on the back burner.

One of the recent changes is that the JSON export gives priority to arrays, even for some properties that don't expose array-style input which would require iterating through wdg otherwise.

Our ideal solution would be support for json representations in opendss itself

The current JSON implementation from DSS C-API is shared in all projects (export only at the moment) and is tightly integrated to the internal property system, which was slowly refactored in DSS C-API through 2018-2021. For JSON, specifically, there are some kinks to handle, but we're almost there. Please take a look at the notebook I linked: https://github.com/dss-extensions/dss_python/blob/master/docs/examples/JSON.ipynb -- I believe we could address some of the remaining issues and reach a good-enough alternative for JSON exports.

After the JSON export gets a stable version, loading JSON directly in the engine shouldn't be too hard. In the original OpenDSS code, parsing and dumping is done separately for each of the DSS classes. In the current DSS C-API, there are some metadata structs and parsing is done in a single file. And the JSON export is mostly done in a single function, for example.

Developed along with the internal JSON functions, there is another alternative which exposes all data directly. This is not ready for prime time yet, but can give you an idea of the possibilities:

• https://github.com/dss-extensions/dss_python/blob/master/tests/test_obj.py
• https://github.com/dss-extensions/dss_python/blob/master/tests/test_batch.py

or at least some kind of 'use strict' export functionality that would get arbitrary .dss file collections into a format that parses easily, reliably and consistently.

In a more general way, that's probably where will end up soon. Both import and export functionality, compatible with DSS-Extensions/AltDSS and the official OpenDSS on Windows. Even if the our engine is not used, it could still be used with a matching official version to facilitate handling data. Indeed there are way too many projects that try to parse OpenDSS files outside the engine, which is a lot of duplicated effort for potentially incomplete results.

I can dream right? (No disrespect intended towards opendss here. We've been writing equally egregious hacks for gridlabd and matpower formats as well.)

Nothing is perfect. Sometimes limitations are known but the priorities don't align, and that's fine, just unfortunate. From my PoV, especially in open-source projects, criticism can be a valid form of contribution and it's rarely meant as disrespect.

[dpinney]

...opendss tutorials... You're probably thinking of the other Paulo who works at EPRI.

My bad!

there's no public issue/bug tracker for OpenDSS

"open" :(

The forum is great though.

AFAIK, there are some variations of OpenDSS in the wild:

It always struck me as odd that all the EPRI material talked about a COM API but opendssdirect worked the same on Windows/Unix. I didn't realize there were so many different opendss implementations!

Developed along with the internal JSON functions, there is another alternative which exposes all data directly. This is not ready for prime time yet, but can give you an idea of the possibilities

I think my takeaway is that I need to take a much closer look at opendsscmd versus opendssdirect and see whether we can migrate functionality back to opendssdirect. Seems like the roadmap you laid out solves a lot of our issues with model translation, and I definitely like the idea of "a lot of bug fixes for Linux, macOS and FPC".

[PMeira]

It always struck me as odd that all the EPRI material talked about a COM API but opendssdirect worked the same on Windows/Unix. I didn't realize there were so many different opendss implementations!

We try to maintain good compatibility where it matters as it was never our idea to fracture the community, which is already small compared to more general OSS projects. Most of the growth so far has been organic, but after some of the next steps, we'll probably try to get some more eyes in our implementation to ensure it is sustainable. Here in Brazil, there were some people stating that DSS-Extensions was closed-source software when we fully embrace open-source solutions (no Delphi required, the current plotting backend is matplotlib, etc.) and most of the distributed binaries are compiled on CI. I guess we need more intro docs in Portuguese...

Seems like the roadmap you laid out solves a lot of our issues with model translation, and I definitely like the idea of "a lot of bug fixes for Linux, macOS and FPC".

On model translation, a whole class of bugs was removed in our version, e.g. this commit to the official version, from yesterday, doesn't affect DSS C-API since that part is fully automated now.

Our first attempt was done by reading opendssdirect's in-memory representation

It seems that was before most of the refactoring branch was merged, so that's good news! One issue that I can see from following the internal data is that it contains all properties, not only those filled by the user. On the current JSON output, that's what DSSJSONFlags.Full controls. For the new API functions I mentioned, there's already a function that returns the internal pointer for that, but it's not used in Python yet: https://github.com/dss-extensions/dss_capi/blob/0.13.2/include/dss_capi.h#L7119-L7127 -- without Full, the JSON output already respects that order.

omf/omf/solvers/opendss/dssConvert.py

Line 76 in 41d6b4f

# hack for malformed wdgcurrents

I noticed some handling of WdgCurrents. That property is read-only and the value is ignored on input. We should probably add a flag to ignore read-only properties and "action"-style properties (properties that have immediate side-effects on other property values but don't make sense to export).

Some properties are already flagged read-only, redundant, or as actions, so its not hard to filter them. If we complement those flags, it should be easier to get a safer, consistent output. We have a document generated from the internal model metadata (which can exported as a big JSON of internal data, by the way) and some flags are already included there. Most ...file= commands can be interpreted as actions since they update the other live properties (e.g. for LoadShapes, Pmult and Qmult are updated, so the external data not required).

omf/omf/solvers/opendss/dssConvert.py

Line 91 in 41d6b4f

# todo: all the settings? which are under dss.Solution and dss.Settings. or don't support them.

The code refactoring I'm finishing next week or so will allow us to automate that for JSON/etc. too 😃

From the newer code:

omf/omf/solvers/opendss/dssConvert.py

Line 97 in 78527ee

ob_data = re.sub(r'zipv=([\d\.\-,]+)', r'zipv=(\1)', ob_data) # HACK: fix zipv with missing parens

ZIPV, for example, is now always shown as an array (if unset it's left zeroed, but you wouldn't see in a save circuit ... output), and all general formatting is consistent -- it doesn't matter if the user used parentheses, quotes, square brackets, etc., the output is already the same.

omf/omf/solvers/opendss/dssConvert.py

Line 105 in 78527ee

#HACK: This is the order in which things need to be inserted or opendss errors out. Lame!

The required order is mostly because the object references are resolved into live pointers on new/edit commands. opendssdirect.Basic.Classes() (DSS.Classes in the COM impl. and DSS-Python) should be enough. There is a classes DSS command too, but I'm not sure what opendsscmd does with its output. I think the older code had the right idea:

omf/omf/solvers/opendss/dssConvert.py

Line 59 in 41d6b4f

all_classes = dss.Basic.Classes()

This quick look at the code you linked already gave me some ideas to enhance the JSON output! I'll try to check the test .dss scripts from OMF in the next few days to see if we still need to fix something there.

[dpinney]

Thank you! We'll dig into the newer versions of opendssdirect and let you know how it goes.

[PMeira]

I ran the sample .dss from the repo and did a save circuit... for each on Windows and most also on Linux (skipped the user-model ones), and I'm happy to confirm that everything seems to be fine nowadays with OpenDSSDirect.py/DSS-Extensions. I left some comments below, but I imagine not all files are used right now, hence the few errors.

In a few weeks, I'll try to finish a more formal JSON proposal after some updates to the current implementation. I can post here again with a link either on this issue or open a more specific one for a request for comments if you prefer to close this.

By the way, just noticed OMF uses NILMTK too. I helped maintain NILMTK for a few years, but due to the pandemic and my workload, I decided to focus on DSS-Extensions for the time being. I hope to contribute more in the future (with Pandas v2, things should be more stable).

---

• The latest OpenDSSCmd on Linux doesn't create the Master.dss correctly (it skips all components). The rest seem mostly OK.

• ieee3.clean.dss: there is an undetected segfault with in the LineCodes using nphases=1 but providing a 3x3 matrix. The official OpenDSS gets an "Access Violation" on Windows. Our current release throws a Python exception warning about the wrong matrix size. OpenDSSCmd silently ignores it.

• ieeeLVTestCase_loadshapeTest.clean.dss: after copying the folder Daily_1min_100profiles from the OpenDSS examples, there was still an error in the line New object=object=Loadshape.Shape_55 npts=1440 minterval=1 mult=(file=Daily_1min_100profiles\Load_profile_55.txt) useactual=true (object=object=). After fixing it, everything matches.

• short_circuit.dss: I'll limit my comments about loading/saving (as the results from this one seem invalid in current versions). For BusCoords, short_circuit_coords.csv was missing, so I just commented the line. These two lines below are invalid and results in errors (on all versions):

New XYCurve.MyPvsT npts=4  xarray=  yarray=
New XYCurve.MyEff npts=4  xarray=  yarray=

The parser interprets the xarray="yarray" and tries to convert the string yarray to a floating-point number, which is not valid (all implementations get this error). Ignoring whether empty arrays make sense for the PVSystem model, something like the following would be valid inputs for the parser:

New XYCurve.MyPvsT npts=4  xarray=[]  yarray=[]
New XYCurve.MyEff npts=4  xarray=""  yarray=""

So, overall everything seems fine:

• There was a minor issue with LoadShape's mean, but it's already fixed and will be included in the next release.
• Not relevant to OMF's usage: one detail is that the official OpenDSS/OpenDSSCmd always includes CalcVoltageBases, which is not always a good idea IMO. I made a note to add a compatibility flag anyway, and we'll probably add a dedicated function for saving the circuit with more general options.

[PMeira]

Hi, @dpinney,
The latest version expose the new Save function with allows some tweaking of the output. There are some changes to try to suppress bogus output, if the elements are edited. This notebook tries to show most of the options: https://dss-extensions.org/OpenDSSDirect.py/notebooks/Saving.html

About the JSON features I mentioned earlier, this is evolving to a formal JSON schema, which we expect will allow unambiguous I/O. There is now whole-circuit FromJSON/ToJSON functions. I'll be working on documenting this, plus adding examples of usage without the DSS engine.

We also added AltDSS-Python, which can be used to complement ODD.py. My team is helping iron some issues there, but most of the functionality is working as expected, exposing all DSS objects in Python and so on.

[dpinney]

Very cool!

We'll definitely take a look and try to use that save to single file feature so we can make this horrible kludge a little nicer.

Ultimately we'd like to move to the JSON interface and get rid of all our regex nonsense. It's very nice to see a formal, versioned schema in something related to opendss. :-)

Modeling .dss syntax as "verb param1=val1 param2=arg2..." is ultimately what we'll stick with for now since that's what we use in our gui but (1) we realize this isn't a great representation, and (2) transforming the opendssdirect JSON to this form will be easy while we think about how to modify our frontend.

[dpinney]

https://dss-extensions.org/OpenDSSDirect.py/notebooks/Saving.html#saving-to-a-single-file

😍