Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

iris.pandas.as_data_frame() n-dimensional Cube conversion #5074

Merged
merged 10 commits into from
Nov 17, 2022
Merged

iris.pandas.as_data_frame() n-dimensional Cube conversion #5074

merged 10 commits into from
Nov 17, 2022

Conversation

trexfeathers
Copy link
Contributor

@trexfeathers trexfeathers commented Nov 16, 2022 •
edited
Loading

🚀 Pull Request

Description

This finally merges @hsteptoe's brilliant work from the pandas_ndim feature branch, allowing as_data_frame() to convert Cubes of any dimensionality to pandas DataFrames, complete with optional dimensional metadata.

Questions

  • Can this be merged as a squash? It's a single self-contained feature that we have collaborated on.
  • If so, should the What's New entry be updated to use the number of this PR (#5074), rather than #4669?
    (Guessing that would mean another tiny PR targeting pandas_ndim).

Consult Iris pull request check list

trexfeathers and others added 7 commits October 24, 2022 11:42
@trexfeathers
Added pandas_ndim FUTURE flag (#4909)
3234489 
* Added pandas_ndim FUTURE flag.

* TODO flag to document pandas_ndim future flag.
@hsteptoe @pre-commit-ci @trexfeathers
Improve iris.pandas cube -> data.frame (#4669)
8744f67 
* Update as_data_frame()

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Minor typo fixes

* Matching data raveling with dimension meshgrid

* Revise test_simple to check long-syle dataframe

* Revise NaN and 1D dataframe tests

* Better pandas.MultiIndex solution

* Add 3D test case

* Fixes for cube with partially defined dims

* Update tests for partially defined dims

* Update time tests

* Reuse _as_pandas_coord()

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Remove Series conversion

* Remove option for copy

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* First go at adding aux coords

* First go at adding global attributes

* Update doc string

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix for time based AuxCoords

* Minor misc fixes

* Fix copy issue

* Fix black weirdness and add copy tests

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add attributes test

* Name fixes

* Add AuxCoord test

* Improved aux coord indexing

* Blacking

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Preserve index on aux coord merge

* Simplify adding AuxCoords

* Add assertRaises test for attributes

* Better dim coord making logic

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Updates to docstring

* Handle multidim AuxCoords

* Add handling for AuxCoords + scalar coord info

* Fix STASH attribute handling

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Improve and simplify dim extraction

* Re-fix copy behaviour

* Doc updates

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Add Series example

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* What's New update

* Add `as_series` depreciation warning

* Fix indent

* flake8 fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Fix deprecation warning

Co-authored-by: Martin Yeo <[email protected]>

* Fix pytest styles

Co-authored-by: Martin Yeo <[email protected]>

* Fix pytest styles

Co-authored-by: Martin Yeo <[email protected]>

* Minor syntax change to list

Co-authored-by: Martin Yeo <[email protected]>

* Update error style

Co-authored-by: Martin Yeo <[email protected]>

* Fix error type

* Reinstate copy warning behaviour

* Add author link

* Remove add_global_attribute code

* Further global attribute code removal

* Add instance checking test

* Improve _make_dim_coord_list efficiency

* Correct list ouput

* Make tests more efficient

* Add scalar coordinate test

* Docstring fixes

* Refactor making of aux_coord_list

* Fix type error

* Add masked array -> nan warning in doc

* Consolidate use of `_as_pandas_coord`

* Roll back breaking _make_dim_coord_list changes

* Raise error for Ancillary variables without dims

* Ancillary variables tweaks

* Ancillary variables tests

* Split out metadata for consistency with `as_cubes`

* Add cell_measure tests

* Docstring fixes

* Test kwarg fixes

* `_make_aux_coord_list` optimisation

* Refactor metadata merging

* Roundabout :issue: fix (remove)

* Roundabout :issue: fix (re-add)

* Docstring typo fixes

Co-authored-by: Martin Yeo <[email protected]>

* Fix doctests

* Update docs/src/whatsnew/latest.rst

Co-authored-by: Martin Yeo <[email protected]>

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

* Further doctest fixes

* [pre-commit.ci] auto fixes from pre-commit.com hooks

for more information, see https://pre-commit.ci

Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
Co-authored-by: Martin Yeo <[email protected]>
@trexfeathers
iris.pandas.as_data_frame() doctests full width (#5057)
c1cb14c 
* iris.pandas.as_data_frame doctests full width.

* NetCDF4 pin to enable RTD build.

* Revert "NetCDF4 pin to enable RTD build."

This reverts commit 31f5ea6.
@trexfeathers @tkknight
Make iris.pandas.as_data_frame() n-dimensional behaviour opt-in (#5059
6443ac5 
)

* Restore original as_data_frame behaviour, with new behaviour as opt-in.

* Document and test opt-in nature of mulit-d as_data_frame behaviour.

* Test fixes.

* Update for Pandas 1.5 deprecation.

* More explicit What's New.

* Fix int64 test.

* added link to the docs archive. (#5064)

* added link to the docs archive.

* added whatsnew

Co-authored-by: tkknight <[email protected]>
@trexfeathers
Move test_pandas into unit tests. (#5071)
c8b99a7
Merge remote-tracking branch 'upstream/main' into update_pandas_ndim
ca0d9fe
@trexfeathers
Merge pull request #5073 from trexfeathers/update_pandas_ndim
f423142 
Update `pandas_ndim` branch from `main`
@stephenworsley
Copy link
Contributor

I think merging without squashing is probably the proper way to go here, I think the only practical difference would be that the blame would point to the individual PRs, but there seems to be enough discussion in those extra PRs that this is worthwhile. I think #4669 is still the most important PR to mention in the whatsnew since it was authored by @hsteptoe, but perhaps you could also add a reference to this PR for completeness. With merging, I think the changes to the whatsnew are probably better done as a PR so that we aren't left with direct commits in the history.

stephenworsley
stephenworsley approved these changes Nov 17, 2022
View reviewed changes
Copy link
Contributor

@stephenworsley stephenworsley left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good job! I think this is all good to merge in now.

@stephenworsley stephenworsley merged commit 1c05e92 into main Nov 17, 2022
@stephenworsley stephenworsley deleted the pandas_ndim branch November 17, 2022 12:25
@trexfeathers trexfeathers mentioned this pull request Nov 17, 2022
Closed
2 tasks
@alxmrs alxmrs mentioned this pull request Mar 2, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@stephenworsley stephenworsley stephenworsley approved these changes

Assignees

@stephenworsley stephenworsley

Labels
Release: Minor Type: Enhancement
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@trexfeathers @stephenworsley @hsteptoe