FETICHE: Framework to import/fetch/transform various aeronautical data

Licensed under the MIT license.

1. About
2. Installation
3. Usage
4. Structure
5. MSRV
6. TODO
7. Contributing

Fetiche is a framework with a set of libraries and utilities dealing with various data formats and import/conversion utilities for Aeronautical data about drones and aircraft.

This is now divided into different crates with libraries (fetiche-engine, fetiche-formats, fetiche-sources) shared by the binary crates (acutectl, opensky-history and now process-data).

Binary crates include command-line utilities (acutectl and opensky-history) to perform import from a file or fetch data from different sites. There is now process-data which include several tasks aimed at gathering statistics and metrics about our drone and flight data.

acutectl is the main data fetching utility, relaying the fetiche-sources and fetiche-formats crates to provide a single interface to multi sources.

process-data works on the drones and flights data through Clickhouse and does various SQL-backed procedures to gather and calculates metrics including distances (2D and 3D).

adsb-to-parquet is a temporary converter between the CSV files we receive the ADS-B data into compressed parquet files. As my patch to improve bdt has been merged, bdt is now used instead.

opensky-history is for retrieving historical data from Opensky. This access is managed through an SSH-based shell to the Impala database. This is for everything older than 1h of real-time data which does complicate things. This utility use the pyopensky Python module (embedded through the inline-python crate).

It might be available at some point as crates on [Crates.io] but for the moment just as a private repository on [GitHub]. Installation can be done either through a compiled binary for your platform or by cloning the repo and compiling.

There is one feature enabled by default, called privacy. This is for truncating the drone ID to a less-easily identifiable value. See Cargo.toml for this.

This is intentionally not a run-time option but a compile-time one.

For the moment, there are 3 binaries called acutectl (with .exe on Windows), opensky-history and process-data. The former is used to fetch data into their native format (csv, json). It uses fetiche-engine for all the code related to accessing, authenticating and fetching data in various ways.

Right now, acutectl use blocking HTTP calls and is not using any async features.

However, while working on streaming support for Opensky, I have been experimenting with tokio for async support and acutectl might eventually become fully-async. It does help for some stuff including signal (read ^C) support.

All the commands are described in more details in the acutectl README.md, opensky-history README.md and process-data files.

On UNIX systems, there is a new command called fetiched. It is a daemon running the latest engine, detaching itself from the terminal and accepting requests through an GRPC interface. The Windows version will have to be run from a specific terminal with the serve command.

In the near future, fetiched is evolving into an Actor-based subsystem (using Actix ) to manage orchestration between the internal modules. We do have an engine actor, a configuration actor, etc.

More details in the specific Fetiched README.md.

NOTE: This is WIP

Each source has its own data model which complicates things, apart from ASTERIX with Cat129 for drone data, each company/service provider use their own data model. To ease managing drone data, I started to define my own DronePoint as a common data model (extracted from the data sent by ASD with some fields with different types -- like actual f32 instead of the string format) and real timestamp. In fact, now that I have fixed Asd struct fields handling and types, it is not needed.

See the fetiche-formats crate for more details.

The Minimum Supported Rust Version is 1.56 due to the 2021 Edition.

• Unix (tested on FreeBSD, Linux and macOS)
• Windows
  • cmd.exe
  • Nushell
  • Powershell (preferred)

Here are some of the things I've been working on. Some of these are registered as issues on GitHub issues.

• support more parameters (like dates, etc.)
• fetch and analyse from Aeroscope
• fetch and analyse from Asd
• divide into crates for sharing more code.
• use a common data model for drone data
• Support for Opensky (same)
• make acutectl use fetiche-engine instead of its own task.rs.
• add streaming support for sources like opensky.
• rename drone-utils into the more proper fetiche-rs.
• Data formats conversion framework
• caching tokens (like ASD ones) locally
• merge import-adsb and cat21conv into acutectl.
• Add a Store module to handle long-running jobs and their output.
• Retrieve historical data from the Opensky site.
• Support for Flightaware AeroAPI and Firehose.
• Apache Parquet as output format.
• Migrate from the embedded DuckDB to a proper server-based DB Clickhouse
• Add more tests & benchmarks.

Uncertain:

• build fetiched as the core daemon and making all other talk to it through gRPC.
• link to HashiCorp Vault for storing credentials and tokens
• support for Safesky for ADS-B data
• Support for Sherlock formats and access methods

Please see CONTRIBUTING.md for some simple rules.

I use Git Flow for this package so please use something similar or the usual GitHub workflow.

1. Fork it here
2. Checkout the develop branch (git checkout develop)
3. Create your feature branch (git checkout -b my-new-feature)
4. Commit your changes (git commit -am 'Add some feature')
5. Push to the branch (git push origin my-new-feature)
6. Create a new Pull Request