Support various datetime types as input #464

seisman · 2020-05-30T00:19:40Z

Description of proposed changes

This PR allows PyGMT accept vectors of various datetime types so that PyGMT can plot datetime axis.

Vectors of following datetime types are supported:

numpy.datetime64
pandas.DateTimeIndex
Raw datetime strings in both ISO and non-ISO formats, e.g., 2010-01-01, 1/1/2018, Jul 5, 2019
Python's built-in datetime and date

TODO:

Wait for the GMT 6.1.0 release
Add more tests
Add a tutorial for calendar axis
Test for xarray.DataArray

Known issues but won't be fixed in this PR:

The region argument doesn't work with numpy datetime64 objects, have to convert to a string using np.datetime_as_string in order to set the map frame bounds.
Support timedelta64, e.g. GPS time from a certain epoch.

Fixes #242.

Example script:

import datetime
import pygmt
import numpy as np
import pandas as pd

fig = pygmt.Figure()
fig.basemap(projection="X15c/5c", region="2010-01-01/2020-01-01/0/10", frame=True)

# numpy.datetime64 types
x = np.array(
    ["2010-06-01", "2011-06-01T12", "2012-01-01T12:34:56"], dtype="datetime64"
)
y = [1.0, 2.0, 3.0]
fig.plot(x, y, style="c0.2c", pen="1p")

# pandas.DatetimeIndex
x = pd.date_range("2013", freq="YS", periods=3)
y = [4, 5, 6]
fig.plot(x, y, style="t0.2c", pen="1p")

# xarray.DataArray
x = xr.DataArray(data=pd.date_range(start="2015-03", freq="QS", periods=3))
y = [7.5, 6, 4.5]
fig.plot(x, y, style="s0.2c", pen="1p")

# raw datetime strings
x = ["2016-02-01", "2017-03-04T00:00"]
y = [7, 8]
fig.plot(x, y, style="a0.2c", pen="1p")

# the Python built-in datetime and date
x = [datetime.date(2018, 1, 1), datetime.datetime(2019, 1, 1)]
y = [8.5, 9.5]
fig.plot(x, y, style="i0.2c", pen="1p")

fig.savefig("datetime.pdf")

Output:

Reminders

Run make format and make check to make sure the code follows the style guide.
Add tests for new features or tests that would have caught the bug that you're fixing.
Add new public functions/methods/classes to doc/api/index.rst.
Write detailed docstrings for all functions/methods.
If adding new functionality, add an example to docstrings or tutorials.

Currently supported datetime types: - [numpy.datetime64](https://numpy.org/doc/stable/reference/arrays.datetime.html) - [pandas.DateTimeIndex](https://pandas.pydata.org/docs/user_guide/timeseries.html)

seisman · 2020-05-30T03:44:49Z

@GenericMappingTools/python @GenericMappingTools/python-contributors This PR is almost done. If you have the GMT latest master branch installed, please try this branch and leave your comments.

weiji14 · 2020-05-31T21:59:26Z

Some instructions on how to test this branch:

curl https://raw.githubusercontent.com/GenericMappingTools/gmt/master/ci/build-gmt-master.sh | bash
conda activate --name pygmt-test  # or some virtual env you have
pip install https://github.com/GenericMappingTools/pygmt/archive/datetime-input.zip

and then in python:

import os

os.environ["GMT_LIBRARY_PATH"] = "/home/username/gmt-install-dir/lib"

import pygmt

pygmt.print_clib_info()  # check that you're using GMT from master

and then try to test out plot with datetimes! I'll have a go at this in a bit with some 'real' data.

weiji14 · 2020-06-01T01:52:01Z

Ok, I tried installing it in a different conda environment and it works now! Below is my convoluted real-data example from Antarctica (longitude=46.522298201524514, latitude=-73.05774246536473).

Code:

# %%
import os

os.environ["GMT_LIBRARY_PATH"] = "/home/username/gmt-install-dir/lib"

import numpy as np
import pygmt

pygmt.show_versions()

# %%
x = np.array(
    [
        np.datetime64(dt)
        for dt in [
            "NaT",
            "2019-01-28T07:51:16.582496785",
            "2019-04-29T03:30:57.620034382",
            "2019-07-28T23:10:36.096639410",
            "2019-10-27T18:50:32.615134843",
            "2020-01-26T14:30:18.835180975",
        ]
    ]
)
y = np.array(
    [np.NaN, 3126.60298909, 3126.67885045, 3126.67276984, 3126.7217197, 3126.68279204,]
)

# %%
# Mask out NaT values
mask = ~np.isnan(y)
x = x[mask]
y = y[mask]

# %%
dates = np.datetime_as_string(x, unit="ns")
region = [dates[0], dates[-1], np.nanmin(y) - 0.1, np.nanmax(y) + 0.1]

# %%
fig = pygmt.Figure()
fig.basemap(
    projection="X15c/5c",
    region=region,
    frame=["WSne", "xaf+lDateTime", "yaf+lHeight(m)"],
)
fig.plot(x=x, y=y, style="t1c", pen="1p")
fig.savefig("height_over_time.png")
fig.show()

Glad to see that these work off the shelf:

Passing in xarray.DataArrays to plot works (thanks to NEP18's __array__ functionality).
ISO datetimes down to nanoseconds are supported by GMT!

Places to improve:

The region argument doesn't work with numpy datetime64 objects, have to convert to a string using np.datetime_as_string in order to set the map frame bounds.
Having even just one NaT value will raise an error, as mentioned in Support datetime data types as input #242 (comment), should be a warning ideally, but it's an upstream issue. Can be tested by commenting out the 'mask' code above.
Support timedelta64, e.g. GPS time from a certain epoch. More of a nice-to-have really, could be done in a separate PR once this datetime one is merged in.

seisman · 2020-06-01T02:51:34Z

Passing in xarray.DataArrays to plot works (thanks to NEP18's __array__ functionality).

I'll add it to the test

ISO datetimes down to nanoseconds are supported by GMT!

Perhaps not true. GMT converts ISO datetime to double values internally. 2019-01-28 07:51:16.582496785 is stored as a double value 1548661876.582496881484985 (printed with %30.15lf). So GMT only supports up to microsecond, as limited by the precision of double type. Maybe I'm wrong here.

The region argument doesn't work with numpy datetime64 objects, have to convert to a string using np.datetime_as_string in order to set the map frame bounds.

The decorator kwargs_to_strings convert the list [w, e, s, n] to a string w/e/s/n. We need to convert datetime64 object to string internally, but it's not ideal to do the conversion in the decorator. I don't have a good solution now.

Having even just one NaT value will raise an error, as mentioned in #242 (comment), should be a warning ideally, but it's an upstream issue. Can be tested by commenting out the 'mask' code above.

Just opened an issue (GenericMappingTools/gmt#3414) in the upstream repository.

Support timedelta64, e.g. GPS time from a certain epoch. More of a nice-to-have really, could be done in a separate PR once this datetime one is merged in.

Perhaps we need to convert timedelta64 to double values before passing to GMT. Leave it for a separate PR.

weiji14 · 2020-06-01T09:15:59Z

ISO datetimes down to nanoseconds are supported by GMT!

Perhaps not true. GMT converts ISO datetime to double values internally. 2019-01-28 07:51:16.582496785 is stored as a double value 1548661876.582496881484985 (printed with %30.15lf). So GMT only supports up to microsecond, as limited by the precision of double type. Maybe I'm wrong here.

Ideally there would be a warning, but I'm not sure if it matters. The data will be plotted in pretty much the same position anyway (unless someone looks at changes every microsecond?) so it's low priority.

Just opened an issue (GenericMappingTools/gmt#3414) in the upstream repository.

Ok, I've tested the 'new' GMT master with GenericMappingTools/gmt#3415 merged in, and it's giving a warning (as expected), but the plot shows up now 🙌

Code (without mask!):

import os

os.environ["GMT_LIBRARY_PATH"] = os.path.join(os.environ["HOME"], "gmt-install-dir/lib")

import numpy as np
import pygmt

pygmt.show_versions()

# %%
x = np.array(
    [
        np.datetime64(dt)
        for dt in [
            "NaT",
            "2019-01-28T07:51:16.582496785",
            "2019-04-29T03:30:57.620034382",
            "2019-07-28T23:10:36.096639410",
            "2019-10-27T18:50:32.615134843",
            "2020-01-26T14:30:18.835180975",
        ]
    ]
)
y = np.array(
    [np.NaN, 3126.60298909, 3126.67885045, 3126.67276984, 3126.7217197, 3126.68279204,]
)

# %%
dates = np.datetime_as_string(x, unit="ns")
region = [dates[1], dates[-1], np.nanmin(y) - 0.1, np.nanmax(y) + 0.1]

# %%
fig = pygmt.Figure()
fig.basemap(
    projection="X15c/5c",
    region=region,
    frame=["WSne", "xaf+lDateTime", "yaf+lHeight(m)"],
)
fig.plot(x=x, y=y, style="t1c", pen="1p")
fig.savefig("height_over_time.png")
fig.show()

…andas.DateTimeIndex and numpy.datetime64 types

seisman · 2020-06-04T05:58:32Z

Passing in xarray.DataArrays to plot works (thanks to NEP18's array functionality).

@weiji14 Could you help provide a simple example for this?

And what's the best place to add some examples for calendar plots?

weiji14

This seems about ready, and it would be nice to merge this before we start working on #520 properly to avoid the conflicts. I'll help to fix the tests in a few minutes, seems to be just whitespace and a wrong error captured.

weiji14 · 2020-07-13T03:05:47Z

pygmt/clib/session.py

+ np.int32: "GMT_INT",
+ np.uint64: "GMT_ULONG",
+ np.uint32: "GMT_UINT",
+ np.datetime64: "GMT_DATETIME",


Quick question, is there a GMT dtype for string types? I can't seem to find one at https://docs.generic-mapping-tools.org/6.1/api.html#gmt-c-api.

There is a GMT_TEXT, but it may not related to the GMT_Put_Strings function.

seisman · 2020-07-13T03:35:09Z

I'll help to fix the tests in a few minutes, seems to be just whitespace and a wrong error captured.

Thanks for help improving this PR. There are still something to do (e.g., add examples, pass a datetime region as a list), but we can merge this PR and open issues for the missing feature.

weiji14

There are still something to do (e.g., add examples, pass a datetime region as a list), but we can merge this PR and open issues for the missing feature.

Yep, we can add those tutorials and enhancements later 🚀

….DataFrame tables (#619) Changes the backend mechanism of `info` from using lib.virtualfile_from_matrix() (which only supports single non-datetime dtypes) to using lib.virtualfile_from_vectors() (which supports datetime inputs as of #464). * Refactor info to use virtualfile_from_vectors to support datetime inputs * Test that xarray.Dataset inputs into pygmt.info works too * Expect failures on test_info_*_time_column on GMT 6.1.1 * Document xarray.Datasets with 1D data_vars as allowed inputs to info Co-authored-by: Dongdong Tian <[email protected]>

Based on GenericMappingTools#464 and GenericMappingTools#549 I prepared a gallery example for plotting datetime inputs.

Support various datetime types as input

6a75bb3

Currently supported datetime types: - [numpy.datetime64](https://numpy.org/doc/stable/reference/arrays.datetime.html) - [pandas.DateTimeIndex](https://pandas.pydata.org/docs/user_guide/timeseries.html)

seisman mentioned this pull request May 30, 2020

Support datetime data types as input #242

Closed

weiji14 mentioned this pull request May 30, 2020

Reusable SpatioTemporal functions weiji14/deepicedrain#40

Merged

4 tasks

Support raw strings and the Python built-in datetime and date

4403a4a

vercel bot temporarily deployed to Preview May 30, 2020 03:16 Inactive

Add more comments

261f93c

vercel bot temporarily deployed to Preview May 30, 2020 03:28 Inactive

seisman added the feature Brand new feature label May 30, 2020

seisman added this to the 0.2.x milestone May 30, 2020

seisman added this to In progress in Release v0.2.x via automation May 30, 2020

Merge branch 'master' into datetime-input

4c60678

vercel bot temporarily deployed to Preview May 31, 2020 22:26 Inactive

This comment has been minimized.

Sign in to view

Merge branch 'master' into datetime-input

34b4aa8

vercel bot temporarily deployed to Preview June 1, 2020 02:55 Inactive

Merge branch 'master' into datetime-input

b9b59a6

vercel bot temporarily deployed to Preview June 1, 2020 23:28 Inactive

Merge branch 'master' into datetime-input

7b78930

vercel bot temporarily deployed to Preview June 4, 2020 04:38 Inactive

Add a function array_to_datetime to convert any legal array/list to p…

fa6ab93

…andas.DateTimeIndex and numpy.datetime64 types

vercel bot temporarily deployed to Preview June 4, 2020 05:41 Inactive

Fix 1D to 1d in docstrings

e21c1ae

vercel bot temporarily deployed to Preview June 4, 2020 06:03 Inactive

Merge branch 'master' into datetime-input

aba4a67

vercel bot temporarily deployed to Preview July 12, 2020 18:58 Inactive

seisman changed the title ~~WIP: Support various datetime types as input~~ Support various datetime types as input Jul 12, 2020

weiji14 reviewed Jul 13, 2020

View reviewed changes

Normalize whitespace to fix doctest in array_to_datetime

607b3ff

vercel bot temporarily deployed to Preview July 13, 2020 03:46 Inactive

Convert to np.datetime64 directly instead of using np.asarray

f4f4015

vercel bot temporarily deployed to Preview July 13, 2020 04:11 Inactive

weiji14 approved these changes Jul 13, 2020

View reviewed changes

Merge branch 'master' into datetime-input

f70db5f

vercel bot temporarily deployed to Preview July 13, 2020 04:19 Inactive

Switch back to using np.asarray, but use dtype 'object' in test

a2258a5

vercel bot temporarily deployed to Preview July 13, 2020 22:47 Inactive

weiji14 marked this pull request as ready for review July 13, 2020 23:22

weiji14 merged commit f401f85 into master Jul 13, 2020

Release v0.2.x automation moved this from In progress to Done Jul 13, 2020

weiji14 deleted the datetime-input branch July 13, 2020 23:51

weiji14 mentioned this pull request Jul 27, 2020

Add tutorial for plotting datetime inputs #549

Closed

4 tasks

This was referenced Aug 10, 2020

Wrap GMT_Put_Strings to pass str columns into GMT C API directly #520

Merged

Support datetime types in region argument #561

Closed

Allow datetime inputs to region argument #562

Merged

weiji14 mentioned this pull request Sep 10, 2020

Running info on pandas.DataFrame with time column doesn't work #597

Closed

weiji14 mentioned this pull request Sep 20, 2020

Refactor info to allow datetime inputs from xarray.Dataset and pandas.DataFrame tables #619

Merged

5 tasks

michaelgrund added a commit to michaelgrund/pygmt that referenced this pull request Jan 1, 2021

Create gallery example for datetime inputs

95c8f63

Based on GenericMappingTools#464 and GenericMappingTools#549 I prepared a gallery example for plotting datetime inputs.

michaelgrund mentioned this pull request Jan 1, 2021

Create gallery example for datetime inputs #779

Merged

seisman mentioned this pull request Apr 5, 2023

Use np.asarray to convert a 1-D array to datetime type in array_to_datetime #2481

Merged

7 tasks

weiji14 mentioned this pull request Dec 3, 2023

pyarrow: Support date32[day] and date64[ms] dtypes in pandas objects #2845

Merged

7 tasks

weiji14 mentioned this pull request Dec 16, 2023

Support timedelta64 dtype as input #2884

Merged

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support various datetime types as input #464

Support various datetime types as input #464

seisman commented May 30, 2020 •

edited

Loading

seisman commented May 30, 2020

weiji14 commented May 31, 2020 •

edited

Loading

This comment has been minimized.

This comment has been minimized.

weiji14 commented Jun 1, 2020 •

edited

Loading

seisman commented Jun 1, 2020

weiji14 commented Jun 1, 2020

seisman commented Jun 4, 2020 •

edited

Loading

weiji14 left a comment

weiji14 Jul 13, 2020

seisman Jul 13, 2020

seisman commented Jul 13, 2020

weiji14 left a comment

Support various datetime types as input #464

Support various datetime types as input #464

Conversation

seisman commented May 30, 2020 • edited Loading

seisman commented May 30, 2020

weiji14 commented May 31, 2020 • edited Loading

This comment has been minimized.

This comment has been minimized.

weiji14 commented Jun 1, 2020 • edited Loading

seisman commented Jun 1, 2020

weiji14 commented Jun 1, 2020

seisman commented Jun 4, 2020 • edited Loading

weiji14 left a comment

Choose a reason for hiding this comment

weiji14 Jul 13, 2020

Choose a reason for hiding this comment

seisman Jul 13, 2020

Choose a reason for hiding this comment

seisman commented Jul 13, 2020

weiji14 left a comment

Choose a reason for hiding this comment

seisman commented May 30, 2020 •

edited

Loading

weiji14 commented May 31, 2020 •

edited

Loading

weiji14 commented Jun 1, 2020 •

edited

Loading

seisman commented Jun 4, 2020 •

edited

Loading