Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add replication file generation and replication from file [WIP] #385

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

simonpoole
Copy link
Contributor

@simonpoole simonpoole commented Jan 28, 2019

This adds a facility to generate replication files in the "classical" OSM directory layout style and a facility to consume such files from a remote location via http.

New options:

Replication files generation:
-replication-dir dir directory for replication files (default '.')
-generate-files generate replication files while updating from Nominatim
-replicate generate a single replication file without starting the server

Consuming replication files:
-replication-url url url of base directory for file based replication (default none)
-replication-interval number interval between trying to read replication files in minutes (default 60)

This is work in progress and will need some testing irl.

This adds a facility to generate replication files in the "classical"
OSM directory layout style and a facility to consume such files from a
remote location via http.
@lonvia
Copy link
Collaborator

lonvia commented Apr 14, 2019

I've played with hourly planet diffs. The resulting files are about half the size of the original diff, which is good.

Looking at the format, we need to get rid of any traces of Nominatim place IDs as they are tied to instances and it must be possible to create a continuous stream of diffs even when the producing instance changes. The ids should be fairly easy to replace. A string of <osm type><osm id><class> uniquely identifies a document. The other fields that contain place ids could preferably be just left out of the dump. They are only intermediate values that are needed to compute the address fields but they are not used in the ES document (although they seem to be imported).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants