Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix DST days in ES parser #6832

Merged
merged 5 commits into from
Jun 28, 2024
Merged

Conversation

silimotion
Copy link
Contributor

Issue

While working on something else, I detected that the ES parser failed for some seemingly random dates. Upon closer inspection, I discovered that this happened because of a failure in the parsing of some dates:

ValueError: Invalid isoformat string: '2020-10-25 1A:00'

Apparently, the REE endpoint uses a strange format to distinguish the repeated hours during DST end days. It seemed quite unusual and I could not find it anywhere, so I built a parser for it.

Preview

An excerpt of the output from the parser after the changes:

 {'correctedModes': [],
  'datetime': datetime.datetime(2020, 10, 25, 0, 55, tzinfo=zoneinfo.ZoneInfo(key='Atlantic/Canary')),
  'production': {'oil': 2.2, 'solar': 0.0, 'wind': 7.8},
  'source': 'demanda.ree.es',
  'sourceType': <EventSourceType.measured: 'measured'>,
  'storage': {'hydro': 4.5},
  'zoneKey': 'ES-CN-HI'},
 {'correctedModes': [],
  'datetime': datetime.datetime(2020, 10, 25, 1, 0, tzinfo=datetime.timezone(datetime.timedelta(seconds=3600))),
  'production': {'oil': 2.9, 'solar': 0.0, 'wind': 6.8},
  'source': 'demanda.ree.es',
  'sourceType': <EventSourceType.measured: 'measured'>,
  'storage': {'hydro': 4.4},
  'zoneKey': 'ES-CN-HI'},
[...]
 {'correctedModes': [],
  'datetime': datetime.datetime(2020, 10, 25, 1, 0, tzinfo=datetime.timezone.utc),
  'production': {'oil': 3.3, 'solar': 0.0, 'wind': 2.9},
  'source': 'demanda.ree.es',
  'sourceType': <EventSourceType.measured: 'measured'>,
  'storage': {'hydro': 1.1},
  'zoneKey': 'ES-CN-HI'},

Behaviour in other parsers

I looked for similar wrong behaviour during these dates in other parsers, in general the situation is quite a mess. In ESIOS and CY parsers it seems the endpoints themselves don't report data for 1 hour. It would be nice if someone with EIA and ENTSOE keys could tests those parsers. Many other parsers only work in realtime so they could not be tested.

Double check

  • I have tested my parser changes locally with poetry run test_parser "zone_key"
  • I have run pnpx prettier@2 --write . and poetry run format in the top level directory to format my changes.

@github-actions github-actions bot added parser python Pull requests that update Python code labels Jun 17, 2024
Copy link
Member

@VIKTORVAV99 VIKTORVAV99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good as far as I can tell but could we get a test for this as well so we don't break it in the future?

@github-actions github-actions bot added the tests label Jun 18, 2024
Copy link
Member

@VIKTORVAV99 VIKTORVAV99 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! 🎉

Thanks for adding the test!

@VIKTORVAV99 VIKTORVAV99 enabled auto-merge (squash) June 28, 2024 11:22
@VIKTORVAV99 VIKTORVAV99 merged commit 245d688 into electricitymaps:master Jun 28, 2024
19 checks passed
@silimotion silimotion deleted the fix-dst-es branch June 28, 2024 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
parser python Pull requests that update Python code tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants