GTFS Support: Transcoding to GraphTiles

[kevinkreiser]

After we have a mock gtfs feed and our tests prove we can load it with the thirdparty library its time to start moving the gtfs data into the tiled format that Valhalla's data processing expects. When we build the routing data we may optionally supply a root directory within which transit data can be found. If that data is present, the routing tile building process will enumerate it and attach it to the regular street graph.

The data format that this process expects to find our own binary GraphTile format but this format is generated from a protobuf encoded one via the valhalla_convert_transit executable. If you are not familiar with protobuf, please familiarize your self with it: https://en.wikipedia.org/wiki/Protocol_Buffers

In brief, we have an existing (though currently semi broken) process for getting transit data into the graph:

1. fetch aggregated transit data from the transitland services json api using the transit_fetcher: https://github.com/valhalla/valhalla/blob/master/src/mjolnir/valhalla_fetch_transit.cc
2. the fetcher tiles this transit data (very similar to gtfs) into the protobuf format defined here: https://github.com/valhalla/valhalla/blob/master/proto/transit.proto (actually transit_fetch.proto but the former is a superset so we can use that)
3. these tiled pbfs are converted into GraphTiles via valhalla_convert_transit
4. the tile build process then attaches those tiles to the regular graph if it finds them in the provided directory

As you can see at the end of the day we need GraphTiles of gtfs data. There are two approaches one could take here:

1. rewrite valhalla_transit_fetcher to, instead of calling transitland, translate gtfs (via just_gtfs) into protobuf transit tiles and then use the rest of the existing code to load the data
2. rewrite valhalla_convert_transit to, instead of converting protobuf transit tiles, create GraphTiles directly from gtfs via just_gtfs

Before starting this task we should look at both avenues and determine the best coarse of action. Once we have decided we can get into the details of how to move the data around.

This task will be complete once:

1. we have a test (we can add it to the previous suite) which, using our faked up gtfs, generates GraphTiles with transit information
2. we have gtest expect/asserts that verify the information is found in the GraphTiles

[pranavpandey1998official]

I am in favour of 2.

1. This would give us more flexibility and room to integrate GTFS-RT and GTFS flex in the future
2. with option 1 we will lose some of the GTFS data (I am not sure about the specifics)

[kevinkreiser]

Just had a quick pairing session with Chris and went over some stuff. We have some notes for the which_tiles implementation in transit_fetcher as follows:

/**
* Here we need to figure out what transit tiles we will eventually build. Because tiles are
 * essentially based on what nodes they contain, in this case stations/stops, all we need to do here
 * is loop over all the feeds in our directory of feeds, load each feed, loop over all the stops in
 * the feed, pull out the lat lon of the stop, and figure out what tile thats in. We can then return a
 * queue as this method already does, and we can even make use of the weighted bit but in this case we
 * would actually need to store rather than just a number as the weight, we'd need to store the feed
 * and the stop_id (or maybe more efficient to get stop by index?). This way when we spawn a bunch of
 * threads, which are burning down tile building tasks, the thread knows for this tile i need to get
 * these stops from these particular feeds. This means that the weighted_tile_t, needs to keep more
 * info, but oh well... To do the intersection of stop ll with tiles we can simply make use of the
 * tilehierarchy object. specifically we need to pull out the level 3 hierarchy (
 * TileHierarchy::GetTransitLevel) and then pull out its "tiles" object. the tiles object has a method
 * tileid method which tells you the index of the tile for the level. so with that you can make a
 * graphid like: GraphId(TileHierarchy::GetTransitLevel().tiles.TileId(ll), 3, 0)
 * @param pt
 * @param feed
 * @return
 */

[kevinkreiser]

fixed in #3669