Adopt NEP29 - drop support for Python 3.6 and NumPy 1.16

[weiji14]

Originally posted at #644 (comment), but updated slightly

Just bringing up the topic of whether we should follow NEP 29 — Recommend Python and Numpy version support as a community policy standard as noted before in #340 (comment). This would mean dropping support for Python 3.6.

In practice, we'll need to follow in the footsteps of our dependent packages:

• pandas 1.2 dropped Python 3.6 at Drop Python 3.6 support pandas-dev/pandas#35214
• xarray is less aggressive, they're thinking about it at Consider revising our minimum dependency version policy pydata/xarray#4179

Some options we can take are:

• Agressive option - Let PyGMT v0.2.x be the last version series to support Python 3.6, and state that PyGMT v0.3.0 requires Python 3.7 or newer.
• Less aggressive option - Still support Python 3.6 for PyGMT v0.3.0 (but maybe drop it from our test suite entirely to save on CI resources).

---

Does it mean that users cannot install pygmt via pip install pygmt or conda install pygmt if they're still using Python 3.6? Or they can still install it, but there is no guarantee that all PyGMT functionalities work well with Python 3.6?

It will be quite hard to support four minor versions of Python (3.6, 3.7, 3.8, 3.9), especially for the conda packages where there is a separate build for each one (unless we can work out how to get a 'noarch' build). We could still allow pip install pygmt Python 3.6 (with no support guarantees, i.e. no CI checks), but it will become a maintenance burden when our dependency packages (numpy, pandas, xarray) move on.

There's no rush to do all this, since Python 3.9 is just released, and there are some libraries still to catch up. Even matplotlib (who has more maintainers) is dropping Python 3.6 for their v3.4 (see matplotlib/matplotlib#17662) so it's worth thinking about following NEP 29 for PyGMT v0.3.0.

Originally posted by @weiji14 in #644 (comment)

Are you willing to help implement and maintain this feature? Yes

[seisman]

As per the comment in #665 (comment), we should strictly follow semantic versioning. As we definitely will add new features in the next few weeks/months, the next release will be v0.3.0, and we can drop the Python 3.6 support now.

In v0.4.0, we will bump the minimum required GMT version to 6.2.0.

[leouieda]

@seisman that depends also on how much work it is to maintain 3.6 support. For 3.5, it was a bit annoying since a lot of pathlib didn't work properly and there were no f-strings. But for 3.6 there isn't much that's like that. So if it's just the extra CI, it might not be too bad. That said, I have no problem with being more aggressive 😡

[seisman]

So if it's just the extra CI, it might not be too bad

We recently had a sudden failure for Windows + Python 3.6 (see #693 and some debugging efforts in #694). I guess something wrong with the conda-forge's Windows builds, but it only happens to Python 3.6. It would be difficult to track down the cause. That's one of the reasons I'd like to drop Python 3.6 support soon.

[weiji14]

A bit of a tangent, but is there a chance we could setup PyGMT to be no-arch (https://www.anaconda.com/blog/condas-new-noarch-packages) like xarray at https://anaconda.org/conda-forge/xarray/files? That would mean that:

1. We don't have to care about what OS or Python version people are using (could be 3.6, 3.9, even the unreleased 3.10 perhaps)
2. The act of adopting NEP29 becomes pretty irrelevant (from PyGMT's point of view), as people would just be forced to update to a newer Python/NumPy version when the time comes.

[seisman]

is there a chance we could setup PyGMT to be no-arch

It looks promising.

The act of adopting NEP29 becomes pretty irrelevant (from PyGMT's point of view), as people would just be forced to update to a newer Python/NumPy version when the time comes.

Dropping Python 3.6 (or any other old Python versions) means that we don't have to run CI jobs for old Python versions, and we can use some new Python features which are only available in new Python versions. So we still need to adopt NEP29 or less aggressive.

[weiji14]

is there a chance we could setup PyGMT to be no-arch

It looks promising.

Ok, PR for noarch Python conda packages is at conda-forge/pygmt-feedstock#13 (generated by conda-forge bot), it looks pretty slick!

[leouieda]

We recently had a sudden failure for Windows + Python 3.6

That's reason enough for me 🙂

And +1 to the noarch thing. Makes the conda-forge builds a lot faster.

[weiji14]

Ok, the noarch conda-forge PyGMT v0.2.1 package is up, and this should work on Python 3.6-3.9 (theoretically on 3.10 too but I haven't tested).

I think we can drop Python 3.6 from our CI (the less aggressive option) now, and decide whether or not we should adopt NEP29 closer to PyGMT v0.3.0's release in a month or two (the aggressive option, i.e. Python 3.6 would just not work at all).

[seisman]

I think we can drop Python 3.6 from our CI (the less aggressive option) now,

I agree!

the aggressive option, i.e. Python 3.6 would just not work at all

How does it work? If we bump the minimum required Python version to 3.7 in setup.py and the conda-forge recipe. What will happen if a Python 3.6 user installs pygmt?

[seisman]

How does it work? If we bump the minimum required Python version to 3.7 in setup.py and the conda-forge recipe. What will happen if a Python 3.6 user installs pygmt?

I tried to install pygmt under Python 3.5:

conda create --name pygmt35 python=3.5
conda activate pygmt35
pip install pygmt

The command installs pygmt v0.2.1 successfully (with some warnings/errors with netcdf4). But when I import pygmt, it gives the following errors, as expected:

>>> import pygmt
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/seisman/.anaconda/envs/pygmtold/lib/python3.5/site-packages/pygmt/__init__.py", line 123
    print(f"  version: {__version__}")
                                    ^
SyntaxError: invalid syntax

As for conda install pygmt, I see the following errors:

Specifications:

  - pygmt -> python[version='>=3.6|>=3.6,<3.7.0a0|>=3.7,<3.8.0a0|>=3.9,<3.10.0a0|>=3.8,<3.9.0a0']

Your python: python=3.5

[weiji14]

The conda case is easy to handle, just need to change recipe/meta.yaml, specifically the runtime requirements at https://github.com/conda-forge/pygmt-feedstock/blob/c3783e707743286ba63d9a32d5fc5f86f87207cc/recipe/meta.yaml#L27

For the pip case, we could strictly enforce Python 3.6+ (or 3.7+ if we adopt NEP29) via a python_requires setting (see https://packaging.python.org/guides/distributing-packages-using-setuptools/#python-requires) in our setup.py file (or pyproject.toml in the future). To be less aggressive (also less work), we could just leave things as it is and folks would just get SyntaxErrors for any non-compatible Python 3.7 code they run (which would be very few at this point).

[seisman]

For the pip case, we could strictly enforce Python 3.6+ (or 3.7+ if we adopt NEP29) via a python_requires setting

That makes sense. I once thought these lines define the minimum supported Python version.

pygmt/setup.py

Lines 31 to 34 in 7e7da78

	"Programming Language :: Python :: 3.6",
	"Programming Language :: Python :: 3.7",
	"Programming Language :: Python :: 3.8",
	"Programming Language :: Python :: 3.9",

Then I'm in favor of the less aggressive option. We only test Python 3.7-3.9, but allow users run pygmt in old Python versions, although there is no guarantee that pygmt will work as expected.

[seisman]

Can we close the issue?

[weiji14]

Should have mentioned this earlier. NEP29 implementation also requires that we set a minimum numpy version (currently numpy=1.15+, will be numpy=1.16+ in Jan 13, 2021).

There's a new PyPI package called nep29 at https://pypi.org/project/nep29/ that might help, see also numpy/numpy#15173. Perhaps we could set up a monthly CI to check for this?

[seisman]

NEP29 implementation also requires that we set a minimum numpy version (currently numpy=1.15+, will be numpy=1.16+ in Jan 13, 2021).

But we're not testing old numpy versions, so how can we know if PyGMT works with numpy 1.15?

[weiji14]

First question is: Do we want to adopt NEP29? If so, then there's a few things we'll need to do.

But we're not testing old numpy versions, so how can we know if PyGMT works with numpy 1.15?

One way is to split our CI into two:

• A 'minimum' (Python 3.6 + NumPy 1.15) check
• A 'maximum' (Python 3.9 + NumPy 1.19) check

This is inspired by xarray's setup at https://github.com/pydata/xarray/tree/v0.16.1/ci/requirements where they have different conda environment.yml files for each setup. There probably aren't many problems with running PyGMT code on NumPy 1.15, but checking a slightly older NumPy version does sound like a good idea to me.

Optionally, we could drop the middle (Python 3.8 + NumPy 1.17) check to save on resources, and have a 'bleeding' edge CI check (Python 3.10+ NumPy 1.20 + GMT Latest) so we're aware of any upstream changes. Again, we don't have to adopt NEP29 straightaway, but it's worth having a think about it.

[seisman]

Is it too much burden for us?

[weiji14]

Ok, some time has passed, and we should now be supporting Python 3.7+ and Numpy 1.17+ only as per NEP29.

Optionally, we could drop the middle (Python 3.8 + NumPy 1.17) check to save on resources,

I think we should do this. It should be relatively straightforward to set the desired numpy version to install after #1033. Currently we have 9 builds in total (3 Python versions x 3 Operating Systems). If we drop the middle Python version, that would save even more CI resources (always feel bad about using up so many CPU cycles!). Specifically we will only need to test (as of March 2021):

• Python 3.7 + Numpy 1.17 + Windows
• Python 3.7 + Numpy 1.17 + macOS
• Python 3.7 + Numpy 1.17 + Linux
• Python 3.9 + Numpy 1.20 + Windows
• Python 3.9 + Numpy 1.20 + macOS
• Python 3.9 + Numpy 1.20 + Linux

I'll work on this NEP29 adoption in the next few hours, should get this sorted for the next PyGMT v0.4.0 release