Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Improve reporting (phase 2) #206

Merged
merged 36 commits into from
Jun 25, 2019
Merged

Improve reporting (phase 2) #206

merged 36 commits into from
Jun 25, 2019

Conversation

khaeru
Copy link
Member

@khaeru khaeru commented Jun 13, 2019 •
edited
Loading

This will partially implement the second phase of the reporting update.

The goal is to have a basic subset of requirements met in message_ix 1.2 / ixmp 0.2 in an experimental state, and then proceed with the remainder for the next release.

Supersedes #176, which was from a branch in a fork; all can commit to this branch.

See also:

Progress
Address requirements from wiki:

  • B1. Provide equal support for any type of model that can be built in MESSAGEix, i.e. do not restrict based on specific features of any one model:
    • e.g. Clara's "hospital" model, i.e. where the 'node' set does not map to countries or geographical regions. deferred; we lack a concrete message_ix.testing function that sets up this model.
    • the Westeros and Austria tutorial models.
    • Single-country models, e.g. South Africa, Israel, others built using Behnam's tools. deferred, ditto.
    • Models with 'extra' features/detail added: MESSAGE-Access, MESSAGE-Transport. deferred, ditto.
    • the MESSAGE global model(s).
      • Basic reporting runs in a cleaned-up version of MESSAGEix-GLOBIOM; see iiasa/message_data#41.
  • B2. Provide zero-config reporting, i.e. reporting that is automatically configured by introspection of the model structure. Including: calculate all outputs of commodities at all levels.
  • B3. Support user-defined operations:
    • B3i. …on exogenous sub-sets of model sets. deferred, requires a specific example, perhaps drawn from iiasa/message_data.
    • B3ii. …that exclude quantities from aggregation operations. deferred, ditto.
    • B3iii. …that compute the same quantity in different ways.
    • B3iv. Cumulative quantities with defined start and end periods. deferred, ditto.
  • B4. Support user-defined reports.
    • B4i. …defined programmatically.
    • B4ii. …loaded from a configuration file. deferred, ditto.
  • B5. Provide (either automatically or per user input) operations related to model features including: deferred, ditto.
    • B5i. …sub-annual time steps.
    • B5ii. …bi-directional commodity flows.
    • B5iii. …user-defined relations. (Probably not automatic; user will need to define reporting.)

PR checklist

  • Tests added (ongoing)
  • Tests merged with Phase 1 tests. N/A, cancelled.
  • Documentation added — preview here.
  • Description in RELEASE_NOTES.md added.

Miscellaneous

  • Choose target release: 1.3 or 2.0? 1.2
  • @gidden w/ assist from @khaeru: add…
    • additional computations for common MESSAGE quantities, e.g. of 'tom = fom + vom.
    • add keys for the 'standard' form of these quantities with keys like 'std:tom'.
    • add a key like 'all:message' that collects these standard quantities.
  • @khaeru update docs to parity with ixmp master.

@khaeru khaeru self-assigned this Jun 13, 2019
@khaeru khaeru mentioned this pull request Jun 13, 2019
Closed
24 tasks
@gidden gidden mentioned this pull request Jun 13, 2019
Closed
5 tasks
This was referenced Jun 13, 2019
Closed
Closed
stickler-ci
stickler-ci reviewed Jun 17, 2019
View reviewed changes
message_ix/testing.py Outdated Show resolved Hide resolved
@gidden gidden mentioned this pull request Jun 19, 2019
Merged
29 tasks
khaeru added 23 commits June 21, 2019 09:45
@khaeru @gidden
add stub class for dask-based reporting
4215882
@khaeru @gidden
draft reporting documentation
c0be867
@khaeru @gidden
disable py27 builds on Travis and Appveyor
462f526
@khaeru @gidden
Add initial tests
26910c3
@khaeru @gidden
Update to sync with ixmp changes
bbd8579
@khaeru @gidden
Report to pyam/IAMC format
9ad11e2
@khaeru @gidden
Write pyam objects to file
b0da521
@khaeru @gidden
Use ixmp.Reporter.full_key
31959d2
@khaeru @gidden
Install pyam & ixmp reporting branch for CI
1f40f70
@khaeru @gidden
Use setuptools.find_packages()
6043dbd
@khaeru @gidden
Appease Stickler
aaeb818
@khaeru @gidden
Install ixmp reporting branch for Appveyor CI
89f2cc0
@khaeru @gidden
Give verbose output for one test failure
035583a
@khaeru @gidden
Include region results in test_report_as_pyam (h/t @danielhuppmann)
cc210b3
@khaeru @gidden
Overload message_ix.Reporter to add common products
26f574f
@khaeru @gidden
Use ixmp.Reporter.add_product()
b8b0c86
@khaeru @gidden
Silence linter
e118849
@khaeru @gidden
Add test_reporter_from_{dantzig,westeros} based on #196
4320e3b
@khaeru @gidden
Pass rename_dims through configure()
aed8c57
@khaeru @gidden
Expand documentation
e8e7ab4
@khaeru @gidden
Add pytest to doc/requirements.txt
2c20e5a
@khaeru @gidden
Appease Stickler/flake8
d93f6a6
@khaeru @gidden
Install ixmp reporting2 branch on RTD
b8bc756
@gidden
Copy link
Member

gidden commented Jun 21, 2019

@khaeru ok with you if I force push a rebased branch?

@khaeru khaeru added enh New features & functionality reporting labels Jun 21, 2019
@khaeru khaeru added this to the 1.2 milestone Jun 21, 2019
@gidden
Copy link
Member

gidden commented Jun 21, 2019

we need to remember to remove CI branch assignment after iiasa/ixmp#150 is merged

@gidden
Copy link
Member

gidden commented Jun 21, 2019

looks like doc making is failing: ImportError: cannot import name 'check_year' from 'ixmp.utils' (/home/travis/build/iiasa/message_ix/doc/src/ixmp/ixmp/utils.py)

@gidden
Copy link
Member

gidden commented Jun 21, 2019

Hmm, this actually did get clobbered with the add_years PR. Is this something you could look into briefly @khaeru ?

@khaeru
Copy link
Member Author

khaeru commented Jun 21, 2019

Hmm, this actually did get clobbered with the add_years PR. Is this something you could look into briefly @khaeru ?

Done.

looks like doc making is failing: ImportError: cannot import name 'check_year' from 'ixmp.utils' (/home/travis/build/iiasa/message_ix/doc/src/ixmp/ixmp/utils.py)

The name check_year was introduced in iiasa/ixmp#142 and so is on the ixmp master branch but not ixmp feature/reporting2. When the latter is merged this should go away.

@khaeru khaeru mentioned this pull request Jun 21, 2019
Merged
@khaeru
Copy link
Member Author

khaeru commented Jun 22, 2019

Tests are passing here now that iiasa/ixmp#150 is merged.

Added two remaining tasks to the PR description.

@gidden
Copy link
Member

gidden commented Jun 22, 2019

Ok, I will take a stab at task one above, likely during travel to Korea tomorrow..

@gidden gidden mentioned this pull request Jun 23, 2019
Merged
@gidden @khaeru
Initial draft of first standard message reporting (#211)
80229bc 
- initial draft of first standard message reporting
- Lint
- Collapse make_westeros_full as an option to make_westeros
- Appease Stickler
- Revert and silence Stickler
- Tidy names, docs; lint
- Avoid repeatedly adding pyam keys
- Combine collapse methods
- Add and use a 'concat' operation for pyam.IamDataFrame
stickler-ci
stickler-ci reviewed Jun 24, 2019
View reviewed changes
tests/test_reporting.py Outdated Show resolved Hide resolved
tests/test_reporting.py Outdated Show resolved Hide resolved
tests/test_reporting.py Outdated Show resolved Hide resolved
@khaeru
Copy link
Member Author

khaeru commented Jun 24, 2019

All greens! We will merge this tomorrow morning unless @danielhuppmann comes up with any critical last-minute TODOs.

@danielhuppmann
Copy link
Member

thanks @khaeru and @gidden for this great new reporting module!

@danielhuppmann danielhuppmann merged commit 3f891dc into master Jun 25, 2019
@khaeru khaeru deleted the feature/reporting2 branch August 9, 2019 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stickler-ci stickler-ci stickler-ci left review comments

@danielhuppmann danielhuppmann Awaiting requested review from danielhuppmann

Assignees

@gidden gidden

@khaeru khaeru

Labels
enh New features & functionality reporting
Projects
None yet
Milestone
1.2
Development

Successfully merging this pull request may close these issues.
4 participants
@khaeru @gidden @danielhuppmann @stickler-ci