Handling of data update delay via DCC is broken

[eknirk]

Describe the bug
The data stored via DCC is delayed, often by many hours. This integration tries to handle at least part of it by still retrieving the previous day's consumption in the first our of the current day. There are two issues with the approach:

1. The implementation is currently broken. The way it is currently implemented, the integration retrieves the data of the previous day for the time between midnight and the current time of the current time, i.e., between a couple of minutes up to an hour.
2. The approach only works when the previous day's consumption is updated during the first hour of the current day. However, this is rarely the case.

While the first point is easy to fix (I have a running patch for it which I am currently testing), the second issue seems impossible to fix. The reason is that home assistant, as far as I know, does not have a way to update past data and relies on the availability of up-to-date data at the time of measurement.

Note that this is not caused by this integration, pyglowmarkt, or homeassistant, but by the data available via DCC. The Bright app and, for example, Octopus suffers from the same delay. To get up-to-date data, it has to be gathered at the smart meter by a CAT, for example Hildebrand's Glow or the Octopus Mini.

In the end, fixing issue (1) will not make much of a difference because data will be lost. Therefore, instead of releasing my patch to fix it, I am asking for opinions and suggestions on how to deal with the delay - so please comment on this issue. One example is Octopus' approach: The Octopus integration does not collect today's consumption data, but only collect's yesterday's data. Using this approach in the energy subsystem allows home assistant to capture more accurate data for long term observation.

[mcc05]

Personally I think there are always going to be issues like this, with data being delayed at capture to use in HA. I would like to see something in HA that allowed a sensor be presented "here's" a data point for this sensor and its from this date/time so add to dataset accordingly and re-calc. Maybe we should ask for a HA enhancement?

I did think about when I was using Octopus data that way always one day out of date, to try a direct SQL insert into the HA database but never got round to it as this integration was working great, maybe that could be another approach?
https://community.home-assistant.io/t/sensor-data-insert-into-existing-table-in-database/406665

[RedEarth1]

Agreed, unless HA ever adds the ability to retrospectively add 'historical' data, I don't think there's much else you can do.

I used to use Domoticz (which does support capture of timestamped 'historical' data) and I created a plugin to retrieve energy usage data from n3rgy.com which is another 3rd party service offering access to smart meter data from the DCC in a similar manner to Hildebrand, but their API only offered usage data for up to the previous day anyway, which sounds similar to the Octopus approach that you mentioned.

Is it possible to make this integration offer multiple consumption sensor entities for each meter, e.g. one that includes todays usage (as it currently does) and another one that only includes yesterday's usage (for those seeking accurate data for long-term analysis), and then leave it for individual users to choose which of these entities to use in their energy dashboards?