-
Notifications
You must be signed in to change notification settings - Fork 30
Transit Data Manager
The Transit Data Manager is the administrative and configuration center for all components. Its main task is to accept all data inputs (save for the real-time Bus Location Data), post-process them as necessary to open formats where appropriate, and make them available to other components.
The TDM has several areas of responsibility:
- API key management
- Centralized Logging
- Bundle Distribution
- Centralized Configuration
- Service Alert Distribution and Persistence
These are accessible via APIs described here.
Transit Data Manager component consists for four modules
- onebusaway-nyc-tdm-webapp - Web application serving as a single point for system configuration, transit bundle deployment etc.
- onebusaway-nyc-tdm-adapters - Contains utility and other classes used by TDM web application.
- onebusaway-nyc-tdm-tcip - Contains JAXB models created from TCIP XML standard.
- onebusaway-nyc-tdm-util - A debug utility to help out with service alerts testing.
First two modules are explained in brief detail.
This module contains classes that serve requests made to TDM web application. It offers web services for providing transit data such as vehicle pullout information, bundle data, crew assignments etc. Inference engine uses data provided by this module to come up with vehicle inference information.
The key classes/services are described below.
- CrewResource - Web service resource that provides crew assignment data. The data is returned as JSON.
- DepotResource - Web service resource that provides list of depots and vehicle assignments for a depot. The data is returned as JSON.
- DscResource - Web service resource that provides destination sign code data. The data is returned as JSON.
- RouteResource - Web service resource for querying routes, specifically route to destination sign code. The data is returned as JSON.
- VehiclePipoResource - Web service resource that provides vehicle pull out data. The data is returned as JSON.
- QrCodeGeneratorResource - Web service resource to generate QR code for a given stop or a list of stops.
- BundleServiceResource - Web service resource that provides bundle data such as list of bundles that can be potentially deployed, bundle files for a particular bundle etc.
- ConfigResource - Web service resource to persist and retrieve configuration values on TDM. The configuration values are stored in a XML file on TDM.
- DeployResource - Web service resource to deploy configuration files staged in S3 to appropriate places on the TDM server
- KeysResource - Web service resource for API key operations such as create, update and delete API keys in the system
- CrewAssignmentDataProviderService - Service used by CrewResource to return crew data. It reads and parses data from the input files uploaded by transit agency.
- DepotDataProviderService - Service used by DepotResource to return depot data. It reads and parses data from the input files uploaded by transit agency.
- VehiclePullInOutDataProviderService - Service used by VehiclePipoResource to return vehicle pullout data. It reads and parses data from the input files uploaded by transit agency.
- VehiclePullInOutService - Provides a list of active vehicle pullouts to VehiclePipoResource.
- GoogleChartBarcodeGenerator - Generates QR Codes using the Google Charts Infographics service.
- BundleDeployer - Deploys bundles to TDM.
- SystemLogResource - Web service resource to provide logging messages to other components of the system. It writes the log messages to obanyc_systemlog table.
- DepotAssignmentPersistenceService - Persists depot data service that links buses to depots.. This service is used by a cron job to persist this data every day.
- DispatchPersistenceService - Persists data such as vehicle pullout data and crew assignment data uploaded by transit agency. This service is used by a cron job to persist this data every day.
- ServiceAlertsResource - Web service resource that lists service alerts of the system.
This module contains services required by TDM web application to return data in the required format. It mostly contains classes that perform data conversion. The data is uploaded in various formats including CSV and XML by the transit agency and is converted into open or well-documented formats such as TCIP.
The key classes/services are described below.
- CrewAssignsInputConverter - Parses crew assignment CSV input files uploaded by the transit agency into custom objects that can be used for TCIP conversion.
- BusDepotAssignsInputConverter - Parses depot assignment XML input files uploaded by the transit agency into custom objects that can be used for TCIP conversion.
- VehicleAssignsInputConverter - Parses vehicle pullout CSV input files uploaded by the transit agency into custom objects that can be used for TCIP conversion.
- MtaDepotMapToTcipAssignmentConverter - Converts depot data provided by transit agency to TCIP format. The TCIP classes are generated in onebusaway-nyc-tcip-api module.
- MtaUtsToTcipAssignmentConverter - Converts UTS crew data provided by transit agency to TCIP format. The TCIP classes are generated in onebusaway-nyc-tcip-api module.
- MtaUtsToTcipVehicleAssignmentConverter - Converts UTS vehicle pullout data provided by transit agency to TCIP format. The TCIP classes are generated in onebusaway-nyc-tcip-api module.
- OperatorAssignmentFromTcip - Converts TCIP operator assignments to JSON model operator assignments. The converted JSON model objects are serialized to get response JSON.
- PullInOutFromTcip - Converts TCIP vehicle pullout data to JSON model vehicle pullout objects. The converted JSON model objects are serialized to get response JSON.
- SignMessageFromTcip - Converts TCIP destination code sign data to JSON model destination sign code objects. The converted JSON model objects are serialized to get response JSON.
- VehicleFromTcip - Converts TCIP vehicle object to JSON model vehicle object.
- SiriServicePersister - Persists service alerts fed to the system in obanyc_servicealert_subscription table.
- NycSiriService - Adapter for receiving service alerts from an external source and integrating them into the Bus CIS Server.
- DepotIdTranslator - Maps datasource abbreviations for depots to a common TCIP abbreviation. Essentially a hash map backed by a CSV file, with pass through methods should the CSV file not be needed/configured.
Service alerts are managed external to the Bus CIS Server. However, their inclusion into the CIS Server is a critical function accomplished by the NycSiriService. The process goes like this:
-
NycSiriServiceClient subscribes to service alerts via an externally provided interface.
-
NycSiriServiceClient processes the results of the SituationExchangeRequest and injects the corresponding service alerts into the Bus CIS Server. This response is authoritative, new service alerts are added, existing service alerts are updated, and missing service alerts are deleted.
-
The new set of service alerts are distributed to front-end clients for display.
The data model corresponding to service alerts functionaly is provided by OneBusAway application modules in the org.onebusaway.transit_data.model.service_alerts package.
Bundles are staged on S3 in <bucket_name>/activebundles/<env>. The Administrative User Interface features The Transit Data Bundle Utility's Deploy Tab which is used to push bundles to the TDM for serving.
The TDM places the bundles in /var/lib/obanyc/bundleservice/activebundles; this directory is polled for updates.
Bundle Conventions: The /api/bundle/list webservice expects the following file structure:
/var/lib/obanyc/bundleservice/activebundles/
<bundle_1>/
BundleMetadata.json
data/
inputs/
outputs/
<bundle_2>/
BundleMetadata.json
data/
inputs/
outputs/
BuneldeMetaData.json is a JSON file of this format:
{
"id": "all",
"name": "Test Bundle From Some Pick",
"applicable-agency-ids": [
"MTA NYCT"
],
"service-date-from": "2011-06-03",
"service-date-to": "2012-04-09",
"created": "2011-11-13T10:10:00-05:00",
"updated": "2011-11-14T10:10:00-05:00",
"files": [
{ "filename": "BaseLocations.txt", "md5": "c1876be140be1e9de079068a1d41fcaf" },
{ "filename": "BlockLayoverIndices.obj", "md5": "bf65d73d8693ab3fc898dd5eeb701851" },
{ "filename": "BlockTripIndices.obj", "md5": "c6d5d375c517b522c3383eec1429ef72" },
{ "filename": "CalendarServiceData.obj", "md5": "907eb39c7fe08efa9139d3946f2c00fd" },
{ "filename": "DSCForTripIndices.obj", "md5": "86da1d78ceceecf2a0681477721d3661" },
{ "filename": "FrequencyBlockTripIndices.obj", "md5": "5a61d172722d8920444274d0249847c7" },
{ "filename": "NarrativeProvider.obj", "md5": "e84613022b293bbb4fa20ae524f71d1f" },
{ "filename": "NotInServiceDSCs.obj", "md5": "ac138569974661654d9114fa10085dcb" },
{ "filename": "NotInServiceDSCs.txt", "md5": "e2fa62c16c0a4160d998dc3ad0756dbe" },
{ "filename": "org_onebusaway_transit_data.log", "md5": "88786eb0aa6170cbc87b269c72cd9a21" },
{ "filename": "org_onebusaway_transit_data.properties", "md5": "914e0b1e70458db2874c4047f256cd95" },
{ "filename": "org_onebusaway_transit_data.script", "md5": "27a2df1991ce186e3508730c1f21919b" },
{ "filename": "ServiceAlerts.xml", "md5": "d41d8cd98f00b204e9800998ecf8427e" },
{ "filename": "ShapeGeospatialIndexData.obj.gz", "md5": "3254e15dc6ae75e2ca730b627cddaca8" },
{ "filename": "TransitGraph.obj", "md5": "136c0e66809b99216c8ce30e054c15a8" },
{ "filename": "TripRunData.obj", "md5": "61908a4ef53a3fedc0b87df1c819885c" },
{ "filename": "TripsForDSCIndices.obj", "md5": "55449f17f5580c87a3e2984f597e5985" },
{ "filename": "RouteSearchIndex/_0.cfs", "md5": "a433ca3d18337c81964a91ef8268e985" },
{ "filename": "RouteSearchIndex/_0.cfx", "md5": "a3a536b54bc4c97770d1861c038ba601" },
{ "filename": "RouteSearchIndex/segments_2", "md5": "8e5d608ee9f93bba75359ac9b17a1f4a" },
{ "filename": "RouteSearchIndex/segments.gen", "md5": "2eb2668d2bf13379fa3584b9a9867b46" },
{ "filename": "StopSearchIndex/_0.cfs", "md5": "52b07d75b51c89fd621be22956643901" },
{ "filename": "StopSearchIndex/_0.cfx", "md5": "67a2d5e0b498631086b5a00dd365fdbe" },
{ "filename": "StopSearchIndex/segments_2", "md5": "e2c5fcc1abf7dc23f8c780938425fb4d" },
{ "filename": "StopSearchIndex/segments.gen", "md5": "2eb2668d2bf13379fa3584b9a9867b46" }
]
}
Note that the property "applicable-agency-ids" is reserved for future use but not actually implemented by the bundle service.
To unify the abbreviations used to represent Bus Depots across the system. Each abbreviation is assocated with its source provider, and ultimately expressed as the abbreviation used by TCIP.
A simple translation tool that uses a single CSV configuration file. Each entry in the configuraiton file will relate a depot ID from one or more data sources to the reference TCIP depot ID.
The tool amounts to a class instantiated by referencing the CSV configuration file, which is then queried by depot and data source name. If no translation is found, the depot abbreviation is passed through unaltered.
DepotIdTranslator (File configFile)
String getMappedId (String dataSourceId, String fromId)
boolean hasMappingForId(String dataSourceId, String fromId)
| DATASOURCE_NAME | FROM | TO |
|---|---|---|
| ABC | JGLT | JG |
| BCE | JG | JG |
An architectural requirement is the ability to add new webservices easily in order to handle new types of data; Spring and Jersey are used for this; An example webservice is here, which listens for RESTful requests at /api/dsc/list (among others), loads data from disk, converts it to TCIP via adapters (SignMessageFromTcip) , then outputs the results via JSON.
In general, the steps to add a new webservice are
- Obtain a data source, be it files copied onto disk, or a web request at some URL
- (optionally) Code an adapter to convert non-standard data to a standards-based format such as TCIP
- Serialize this data to JSON
- Code a Resource file (like the one linked to above) that defines the endpoint and links the above steps together.