Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geocoding 2.0 #304

Merged
merged 79 commits into from
Jan 29, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
79 commits
Select commit Hold shift + click to select a range
173d618
Put parents into regions dataframe and geodataframe
IKupriyanov-HORIS Sep 11, 2020
c8f8d1b
Implementing parents in regions_builder.
IKupriyanov-HORIS Sep 14, 2020
1fccca9
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Sep 14, 2020
6fef609
WIP: parents
IKupriyanov-HORIS Sep 15, 2020
9b5ae98
Fix tests, sync protocol with server
IKupriyanov-HORIS Sep 15, 2020
38f1db7
Fix more tests
IKupriyanov-HORIS Sep 16, 2020
54b4457
Fix more tests
IKupriyanov-HORIS Sep 25, 2020
299b6aa
New server IP in demo
IKupriyanov-HORIS Sep 25, 2020
701f78c
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Oct 5, 2020
a37a3f5
WIP: answers in protocol
IKupriyanov-HORIS Oct 14, 2020
f6ca8a3
WIP: moving to Answers instead of flat list of GeocodedFeature
IKupriyanov-HORIS Oct 14, 2020
52e7c1f
WIP: link between request and response
IKupriyanov-HORIS Oct 15, 2020
be18cc5
Getting rid of a query string from Answer and GeocodedFeature
IKupriyanov-HORIS Oct 17, 2020
e8219a8
Remove query from Answer and GeocodedFeature
IKupriyanov-HORIS Oct 19, 2020
326550b
Fix assertion condition
IKupriyanov-HORIS Oct 20, 2020
f6fc7c8
Remove chunked requests, better docs for new API
IKupriyanov-HORIS Oct 20, 2020
cbc1745
WIP: scope
IKupriyanov-HORIS Oct 21, 2020
02aa0ba
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Oct 21, 2020
cca501d
Update example with counties
IKupriyanov-HORIS Oct 22, 2020
91b027a
Update example with counties
IKupriyanov-HORIS Oct 22, 2020
21ec386
Basic support for scope in the new geocoding API
IKupriyanov-HORIS Oct 22, 2020
6f99bbe
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Oct 23, 2020
60968ca
Default scope value to None
IKupriyanov-HORIS Oct 26, 2020
f44d979
Unit tests for new geocoding API
IKupriyanov-HORIS Oct 28, 2020
9ed17b0
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Oct 28, 2020
539ef98
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Oct 28, 2020
4345821
Request validation, tests
IKupriyanov-HORIS Oct 28, 2020
648eef7
Experiment with map and nbviewer
IKupriyanov-HORIS Oct 29, 2020
d492167
Experiment with map and nbviewer and stamen tiles
IKupriyanov-HORIS Oct 29, 2020
bc846fa
Better parents/scope support, more tests
IKupriyanov-HORIS Oct 30, 2020
dcfd17b
where(..., near=..., within=...)
IKupriyanov-HORIS Nov 2, 2020
41e0ce1
Better error handling
IKupriyanov-HORIS Nov 3, 2020
26804b4
Tests for not available features in new API
IKupriyanov-HORIS Nov 4, 2020
fbd7c50
Better error handling for where function (missing key, scope and othe…
IKupriyanov-HORIS Nov 5, 2020
c444036
map_join with multikeys
IKupriyanov-HORIS Nov 14, 2020
0e213d9
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Nov 14, 2020
f7f5df8
Single key data_join_on and map_join_on.
IKupriyanov-HORIS Nov 15, 2020
a31ca74
Multi key data_join_on and map_join_on.
IKupriyanov-HORIS Nov 16, 2020
1f69608
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Nov 23, 2020
08f250f
map_join with dups, but livemap fails with null values in data
IKupriyanov-HORIS Nov 24, 2020
71060c2
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Nov 25, 2020
6c5f57b
Handle null values in pie/bar
IKupriyanov-HORIS Nov 26, 2020
cec1bdc
LP-62
IKupriyanov-HORIS Nov 27, 2020
c5375fb
Better error message on using scope with parents
IKupriyanov-HORIS Dec 4, 2020
5a2569d
Better error message for invalid scope type, countries request suppor…
IKupriyanov-HORIS Dec 7, 2020
4926246
More tests for us-48
IKupriyanov-HORIS Dec 8, 2020
c7c72ba
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Dec 8, 2020
dc5a21f
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Dec 14, 2020
b0ba74f
WIP: changing public API
IKupriyanov-HORIS Dec 15, 2020
536ed25
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Dec 18, 2020
fe750f7
Support Geocoder as geom_xxx(map=...) parameter
IKupriyanov-HORIS Dec 21, 2020
c317a16
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Dec 21, 2020
7531be1
Code cleanup, more tests
IKupriyanov-HORIS Dec 22, 2020
425fef3
Copy not matched dups from map on map_join (fix for last cell in geop…
IKupriyanov-HORIS Dec 23, 2020
ed5f0ea
Use single entry parent for all names
IKupriyanov-HORIS Dec 23, 2020
371525a
Fix empty result for select all kind of request
IKupriyanov-HORIS Dec 24, 2020
481b90d
where -> scope, near -> closest_to, single entry scope, remove auto-c…
IKupriyanov-HORIS Jan 11, 2021
4f3e114
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Jan 11, 2021
5002f32
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Jan 11, 2021
d316cbe
Remove MapRegion from 'scope' type error message
IKupriyanov-HORIS Jan 13, 2021
bfe2d0b
scope can now be used with county and state. scope and country will c…
IKupriyanov-HORIS Jan 14, 2021
f1a3014
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Jan 18, 2021
e747a7e
Handle any Iterable types in parents functions (counties/states/count…
IKupriyanov-HORIS Jan 20, 2021
3915dba
Enable gzip for geocoding responses
IKupriyanov-HORIS Jan 21, 2021
d8b1fac
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Jan 21, 2021
93c7aaa
Update docs
IKupriyanov-HORIS Jan 21, 2021
f1e98cd
Update docs
IKupriyanov-HORIS Jan 21, 2021
4b4a9ba
Keep original query name for ambiguous result with allow_ambiguous flag
IKupriyanov-HORIS Jan 22, 2021
ffaaab8
Fix error message
IKupriyanov-HORIS Jan 22, 2021
50156ed
Remove method to_data_frame and interface CanToDataFrame
IKupriyanov-HORIS Jan 25, 2021
8131764
WIP: update geocoding.md
IKupriyanov-HORIS Jan 26, 2021
1917b39
WIP: update geocoding.md
IKupriyanov-HORIS Jan 26, 2021
a908666
Revert changes to builder.ipynb
IKupriyanov-HORIS Jan 26, 2021
ff331eb
WIP: update geocoding.md
IKupriyanov-HORIS Jan 26, 2021
b6fa9b4
Use geocoding level instead of 'request' for a column name, remove da…
IKupriyanov-HORIS Jan 28, 2021
851fa40
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Jan 28, 2021
83002eb
Support Geocoder in ggplot(data=...)
IKupriyanov-HORIS Jan 29, 2021
99bd3c3
Switch to the new geocoding server
IKupriyanov-HORIS Jan 29, 2021
a203c3b
Merge branch 'master' into geoservices-1.1.0
IKupriyanov-HORIS Jan 29, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Keep original query name for ambiguous result with allow_ambiguous flag
  • Loading branch information
IKupriyanov-HORIS committed Jan 22, 2021
commit 4b4a9ba589d61d38ba4e94caf51c4bb91354909a
25 changes: 16 additions & 9 deletions python-package/lets_plot/geo_data/geocodes.py
Original file line number Diff line number Diff line change
Expand Up @@ -43,10 +43,17 @@ class Resolution(enum.Enum):
world_low = 1


def select_request(query: RegionQuery, answer: Answer, feature: GeocodedFeature) -> str:
# exploding answers (features count > 1) don't have exact request (like us-48, it can't be a proper
# request for 48 features/states) and so feature name should be used as request.
return query.request if len(answer.features) <= 1 else feature.name
def select_request_string(request: Optional[str], name: str) -> str:
if request is None:
return name

if len(request) == 0:
return name

if 'us-48' == request.lower():
return name

return request


def zip_answers(queries: List, answers: List):
Expand All @@ -64,9 +71,9 @@ def __init__(self):
self._state: List[Optional[str]] = []
self._country: List[Optional[str]] = []

def append_row(self, request: str, found_name: str, query: Optional[RegionQuery]):
self._request.append(request)
self._found_name.append(found_name)
def append_row(self, query: RegionQuery, feature: GeocodedFeature):
self._request.append(select_request_string(query.request, feature.name))
self._found_name.append(feature.name)

if query is None:
self._county.append(MapRegion.name_or_none(None))
Expand Down Expand Up @@ -138,7 +145,7 @@ def to_map_regions(self) -> List[MapRegion]:
regions: List[MapRegion] = []
for answer, query in zip_answers(self._answers, self._queries):
for feature in answer.features:
regions.append(MapRegion.place(feature.id, select_request(query, answer, feature), self._level_kind))
regions.append(MapRegion.place(feature.id, select_request_string(query.request, feature.name), self._level_kind))
return regions

def as_list(self) -> List['Geocodes']:
Expand Down Expand Up @@ -286,7 +293,7 @@ def to_data_frame(self) -> DataFrame:
# for us-48 queries doesnt' count
for query, answer in zip_answers(self._queries, self._answers):
for feature in answer.features:
places.append_row(select_request(query, answer, feature), feature.name, query)
places.append_row(query, feature)

data = {**data, **places.build_dict()}

Expand Down
10 changes: 5 additions & 5 deletions python-package/lets_plot/geo_data/gis/response.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,11 +38,11 @@ def __init__(self, geometry: Union[Multipolygon, Polygon, GeoPoint]):
class GeocodedFeature:
def __init__(self,
id: str, name: str,
highlights: Optional[List[str]],
boundary: Optional[Boundary],
centroid: Optional[GeoPoint],
limit: Optional[GeoRect],
position: Optional[GeoRect]):
highlights: Optional[List[str]]=None,
boundary: Optional[Boundary]=None,
centroid: Optional[GeoPoint]=None,
limit: Optional[GeoRect]=None,
position: Optional[GeoRect]=None):
assert_type(id, str)
assert_type(name, str)
assert_optional_list_type(highlights, str)
Expand Down
10 changes: 5 additions & 5 deletions python-package/lets_plot/geo_data/to_geo_data_frame.py
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,9 @@
from pandas import DataFrame
from shapely.geometry import box

from lets_plot.geo_data import PlacesDataFrameBuilder, zip_answers, select_request, DF_REQUEST, DF_FOUND_NAME, abstractmethod
from lets_plot.geo_data.gis.response import Answer, GeocodedFeature, GeoRect, Boundary, Multipolygon, Polygon, GeoPoint
from lets_plot.geo_data import PlacesDataFrameBuilder, zip_answers, select_request_string, abstractmethod
from lets_plot.geo_data.gis.request import RegionQuery
from lets_plot.geo_data.gis.response import Answer, GeocodedFeature, GeoRect, Boundary, Multipolygon, Polygon, GeoPoint

ShapelyPoint = shapely.geometry.Point
ShapelyLinearRing = shapely.geometry.LinearRing
Expand Down Expand Up @@ -48,7 +48,7 @@ def to_data_frame(self, answers: List[Answer], queries: List[RegionQuery]) -> Da
for feature in answer.features:
rects: List[GeoRect] = self._read_rect(feature)
for rect in rects:
places.append_row(request=select_request(query, answer, feature), found_name=feature.name, query=query)
places.append_row(query, feature)
self._lonmin.append(rect.min_lon)
self._latmin.append(rect.min_lat)
self._lonmax.append(rect.max_lon)
Expand Down Expand Up @@ -85,7 +85,7 @@ def to_data_frame(self, answers: List[Answer], queries: List[RegionQuery]) -> Da

for query, answer in zip_answers(queries, answers):
for feature in answer.features:
places.append_row(request=select_request(query, answer, feature), found_name=feature.name, query=query)
places.append_row(query, feature)
self._lons.append(feature.centroid.lon)
self._lats.append(feature.centroid.lat)

Expand All @@ -103,7 +103,7 @@ def to_data_frame(self, answers: List[Answer], queries: List[RegionQuery]) -> Da
geometry = []
for query, answer in zip_answers(queries, answers):
for feature in answer.features:
places.append_row(request=select_request(query, answer, feature), found_name=feature.name, query=query)
places.append_row(query, feature)
geometry.append(self._geo_parse_geometry(feature.boundary))

return _create_geo_data_frame(places.build_dict(), geometry=geometry)
Expand Down
55 changes: 48 additions & 7 deletions python-package/test/geo_data/test_geocoder.py
Original file line number Diff line number Diff line change
Expand Up @@ -10,9 +10,9 @@
from lets_plot.geo_data.gis.geometry import GeoRect, GeoPoint
from lets_plot.geo_data.gis.request import MapRegion, AmbiguityResolver, GeocodingRequest, LevelKind, RegionQuery, \
IgnoringStrategyKind
from lets_plot.geo_data.geocoder import geocode_countries, geocode, NamesGeocoder, geocode_cities
from lets_plot.geo_data.geocoder import geocode_countries, geocode, NamesGeocoder, geocode_cities, geocode_states
from lets_plot.geo_data.geocodes import Geocodes
from .geo_data import make_answer
from .geo_data import make_answer, assert_row
from .request_assertion import GeocodingRequestAssertion, QueryMatcher, ScopeMatcher, ValueMatcher, eq, empty, \
eq_map_region_with_name, eq_map_region_with_id

Expand Down Expand Up @@ -153,11 +153,7 @@ def test_where_closets_to_point():
GeocodedFeature(
id='foo_id',
name='foo',
highlights=None,
boundary=None,
centroid=GeoPoint(1, 2),
limit=None,
position=None
centroid=GeoPoint(1, 2)
)
])
]
Expand Down Expand Up @@ -192,6 +188,51 @@ def test_select_all_query_with_empty_result_should_return_empty_dataframe():
assert 0 == len(centroids)


@mock.patch.object(GeocodingService, 'do_request', lambda self, reqest: SuccessResponse(
message='',
level=LevelKind.state,
answers=[
Answer(
features=[
GeocodedFeature(id='foo_id', name='foo'),
GeocodedFeature(id='bar_id', name='bar'),
GeocodedFeature(id='baz_id', name='baz'),
]
)
]
))
def test_for_us48_request_should_contain_feature_name():
states = geocode_states('us-48')

assert_row(
states.get_geocodes(),
names=['foo', 'bar', 'baz'],
found_name=['foo', 'bar', 'baz']
)


@mock.patch.object(GeocodingService, 'do_request', lambda self, reqest: SuccessResponse(
message='',
level=LevelKind.city,
answers=[
Answer(
features=[
GeocodedFeature(id='foo1_id', name='Foo'),
GeocodedFeature(id='foo2_id', name='Foo'),
GeocodedFeature(id='foo3_id', name='Fooo'),
]
)
]
))
def test_allow_ambiguous_result_should_keep_request():
cities = geocode_cities('foo')

assert_row(
cities.get_geocodes(),
names=['foo', 'foo', 'foo'],
found_name=['Foo', 'Foo', 'Fooo']
)


def test_allow_ambiguous():
request = geocode(names='foo')\
Expand Down
9 changes: 8 additions & 1 deletion python-package/test/geo_data/test_integration_new_api.py
Original file line number Diff line number Diff line change
Expand Up @@ -278,4 +278,11 @@ def test_counties():

geocoded_counties = geodata.geocode_counties(counties).states(states).scope('usa').get_boundaries('country').request

assert counties == geocoded_counties.tolist()
assert counties == geocoded_counties.tolist()


def test_request_in_ambiguous_df():
us48 = geodata.geocode_states('us-48').get_geocodes()
warwick = geodata.geocode_cities('warwick').allow_ambiguous().get_geocodes()

assert_row(warwick, names='warwick', found_name='Warwick')