Middleware?

[faultyserver]

Middleware would work similarly to Rack middleware, enabling runtime-configured event handling and data modification.

One use case for this feature would be to associate Stops and Vehicles to Routes. While this could also be done in Source#update, doing so in middleware would avoid the need to couple different Source instances together or have cache storage to perform object lookups.

[elliottwilliams]

Would this also make it relatively easy to support features not "core" to transit agencies? e.g. saturation, neighborhood geofences, etc

[faultyserver]

Eh, not really. But the thought behind why is kind of complicated. I also pretty quickly changed my mind about where association should happen per #2, both of which I'll try to explain here.

Design

There are essentially three layers of responsibility in the system:

• Sourcing: loading data from different sources (duh) and normalizing it to a consistent data structure (renaming fields, choosing primary keys, etc.).
• Management: determining an "active set" of objects, persisting data (see Modular Object Storage #2), scheduling updates, triggering events.
• Publishing: sending events over the network (just WAMP for now), logging, error handling (errors will be published as events), etc.

Middleware essentially generalizes the Publishing layer, creating an upward growing stack. This is a little backwards compared to Rack, where middleware is considered "below" the application:

Shark:                            Rack:
------                            -----
...             ∧                 Rails/Sinatra
WAMP Transport  ∧              ∨  Logger
Error Handler   ∧- Middleware -∨  Error Handler
Logger          ∧              ∨  HTTP Transport
Management                     ∨  ...
Sourcing                          ...

I think of it this way because it better mirrors an Operating System. Sourcing like the kernel, dealing mostly with single objects and modifying their properties. Management is the standard library, dealing with multiple objects and triggering events. Middleware/Publishing is the user space, where interaction with the system can occur. Rack can use the same metaphor with its stack, it just happens to work the other way around.

Solutions

Now, the principle behind why "core shimming" wouldn't really work in middleware: everything from Management up should be idempotent. Creating associations and shimming attributes both violate that principle.

The solution is to do these things in the Sourcing layer. Since Sources are pretty much agency-specific already, any agency that doesn't provide an attribute can shim that attribute when creating the data object in the Source.

For creating associations, a consequence of #2 means that Sources will have access to all objects, and associations can be created by polling the data storage adapter. Here, the associations should not embed objects, but rather include just the unique identifier of the associated object. Then, Middleware classes will also be able to poll the data storage adapter to embed objects when necessary. This reduces memory costs for highly-connected objects like Routes with Stops.

The data storage adapter is thus a transient module (not directly tied to any layer), acting as a static information mediary that can be guaranteed to exist, and does not need to passed around.

[faultyserver]

tl;dr for all of that: Middleware is essentially a future-proofing addition for now, in case we decide to do more than just WAMP communications at some point.

[faultyserver]

Technically closed via #3, but I'll leave this open for now until the stack-like system gets implemented.