postprocessor translation

[jdherman]

Based on meeting with @mfefer let's try to keep track of postprocessor features that we need. Here is a list of the current Excel postprocessors, with my understanding of what they do, how important they are, and how close we are to having those features written in python between postprocess.py and aggregate_regions.py. Please edit or correct anything that seems off.

• Ag Post Processor: deliveries, shortage volume & cost, and marginal cost at each demand node. This is all done in postprocess.py except the marginal cost, which will be an important feature to add. Possibly by separating the economic stuff into a separate script as suggested by @msdogan.

• Ag Supply Portfolio: breaks down the % of supply type for each region. @mfefer is working on this now. Will be important to compare to previous work.

• Basin Outlet Flow Comparison: This looks like it compares flows in CALVIN with CALSIM. I hadn't seen this before. Was this just for comparison purposes?

• Debug Flow Finder: As the name suggests. We already have this written.

• Desal Plants: Probably don't need this right now for CEC. Especially if desal is already included in the urban/ag portfolio calculations.

• Flow Finder: Extract flow time series for particular link(s). We already have this ability by running postprocess.py and then filtering the resulting flow.csv file with pandas.

• GW Recharge: Not sure what this does. Is this based on amplitudes?

• GW Storage: We already get the GW storage time series from postprocess.py.

• Hydropower: Will be interesting to do this later, maybe not an immediate goal. Needs to somehow take monthly flow values out of reservoirs and estimate the amount of energy generated in the process.

• MIF_Marginals: Specific to environmental flow constraints. We already have these dual values, and is probably not a near-term focus.

• Operating Costs: don't worry about for now

• Pumping: not now

• Refuge marginals: not now

• SW_Evap: Max already did this in postprocess.py

• SW_Pumping: not now ?

• SW Storage Marginals: we already have this data

• SW Storage Postprocessor: this just extracts the time series, we have this

• Urban postprocessor: deliveries, shortage volume & cost, and marginal cost for urban delivery links (same as ag). We already have this.

• Urban supply portfolio: Max is working on this now.

Reading through the Excel postprocessors, it seems like the most valuable information is the pathnames. (Which links are SW vs. GW vs. environmental ... etc.) We should think about how to preserve this somehow.

Right now I think the priorities are the portfolios. Then we can compare data with past CALVIN runs as soon as we get it working without debug mode. Thanks!

[msdogan]

@jdherman I think two important postprocessors are ag and urban postprocessors.

• Ag and urban supply portfolio postprocessors can be combined with ag and urban postprocessors if possible. It does not have to be separate.
• We shouldn't worry about Basin Outlet Comparison. (I agree)
• Finding flow/duals by filtering time-series will be really helpful. Maybe more than other postprocessors.
• GW recharge is not that important.
• Hydropower will require some work, storage, release, their capacities, price and some other data. I agree it is not a priority.
• And remaining points sound good to me :)

Yes, we used those pathnames to filter from the HEC-DSS database. Similarly, we now have link names and some other identifiers. We have to update those if we make any changes to network that affect postprocessors. Not sure if there is any automated way to keep those up-to-date.

[jdherman]

Note for the operating costs / hydropower postprocessors: the model should save the final links.csv file (after reducing LBs in debug mode). And for limited foresight it will need to save all of the annual files.

This will let us look at what debug mode is reducing in an automated way ... and will also make it easier to run the postprocessors without rebuilding the model.

(Per meeting with Max 9/14/17)