Running nlcd_bycoords for many coordinates

[ablythed]

Hello! In my spare time, I've been learning about natural resource datasets, and stumbled upon hyriver - what an awesome project. Thanks to all the contributors for putting this together!

I'm interested in inputing a list of coordinates representing various observation locations for a species and outputting the land use info (impervious, cover, canopy, and descriptor) from the NLCD database. The list for the species I'm using as an example has about 3800 entries, but ideally I would like to do the same thing for much longer lists too. I've been using nlcd_bycoords, but after iterating through about 1500 coordinates, which took a little over 20 min, it errored out, saying the connection was reset. Am I doing something wrong, or is this expected?

I tried iterating over the list lookup, which contains my coordinates and looks something like [[-80.874139, 25.193481], [-81.202958, 29.542953], [-82.640959, 28.947379], [-82.640854, 28.947255], [-82.640655, 28.947388], [-81.017561, 28.536389],... as well as just inputting the entire list, like this:

output = gh.nlcd_bycoords(coords=lookup)

and

output = gh.nlcd_bycoords(coords=lookup[0])
for i in range(len(lookup)):
    output = pd.concat([output,gh.nlcd_bycoords(coords=lookup[i]])

The output I did get from the second one before it errored out was exactly what I was looking for.

Thanks in advance - I am a beginner with this package and appreciate any help.

[cheginit]

Hi! The issue with large requests is that it hammers the web service and can lead to the service blocking the request. I recommend an approach similar to your for-loop but submitting your requests in batches, instead. For example, for a batch size of 100, you do something like this:

from pathlib import Path
import cytoolz as tlz

for i, c in enumerate(tlz.partition_all(100, lookup)):
    if not Path(f"nlcd_{i}.parquet").exists():
        gh.nlcd_bycoords(coords=c).to_parquet(f"nlcd_{i}.parquet")

Whenever it fails, you can rerun it again, and it will continue from where it failed. You can run this a couple of times, until it finishes. Once it's done, you can just read and concat all the files into a single dataframe.

[ablythed]

Thank you so much!