v0.1.2: new core

[cathaypacific8747]

The current core is quite cumbersome to use:

async with FR24() as fr24:
    fl = fr24.flight_list("id")
    fl.data.load() # read flight_list/{id}.parquet
    response = await fl.api.fetch() # optional parameters goes here, e.g. page, limit
    fl.data.add(response) # json -> pyarrow
    fl.data.save() # write flight_list/{id}.parquet

It is unclear where data is actually stored. It would be ideal to have something like:

async with FR24() as fr24:
    await fr24.login()
    data_old = fr24.flight_list.load("id") # reads flight_list/{id}.parquet
    # FlightListArrow { ctx: FlightListContext, table: pyarrow.Table }

    data_new = (
        await fr24.flight_list.fetch("id") # FlightListAPIResp { ctx: {"id": "{id}", "page": "..."}, response: json }
    ).to_arrow() # FlightListArrow { ctx: ..., table: pyarrow.Table }
    data = data_old.concat(data_new)
    data.save() # calls self.table.write_parquet("flight_list/{id}.parquet")

• make logins explicit instead of trying to login in __aenter__
• wrap around the raw/processed data with composition

Things to implement for the new core:

• .flight_list.fetch_all()
• .playback.*
• .livefeed.*
• .find.*
• store request context in arrow table .schema.metadata

[cathaypacific8747]

The rewrite is now complete, the core now supports three major services (livefeed, playback and flight_list):

async def main() -> None:
    async with FR24() as fr24:
        # download livefeed of entire world
        response = await fr24.livefeed.fetch() # {ctx: {...}, data: list[dict]}
        datac = response.to_arrow() # {ctx: {...}, data: pa.Table}
        datac.save() # saves to ~/.cache/fr24/livefeed/{timestamp}.parquet, determined from .ctx
        datac_disk = fr24.livefeed.load(timestamp) # reads data from cache and builds ctx

        # download and save trajectory for one flight
        response = await fr24.playback.fetch(0x2FB3041).to_arrow().save()
        datac_disk = fr24.playback.load(0x2FB3041)
        print(datac_disk.metadata) # accesses metadata stored in arrow schema

        # iteratively upsert pages of flight list to an existing arrow table
        data = fr24.flight_list.load(reg="B-HPB")
        async for response in fr24.flight_list.fetch_all(reg="B-HPB"):
            data_new = response.to_arrow()
            data.concat(data_new, inplace=True)
            data.save()

await main()

Documentation and tests are also updated for the new core. Some details:

• low level functions are left mostly untouched, except:
  • login_with_token_subscription_key: checks if the token is expired by parsing the JWT claim
  • livefeed_flightdata_dict: added missing squawk field
  • find: parameterised AsyncClient and modified parts of the TUI to use the app state's one instead
• switched to httpx.AsyncClient(http2=True) by default to avoid 464/525
• CLI now supports all three services
• previously, protobuf definitions are heavily nested - they have been rewritten to their match the official names and structure as far as possible
• fr24 seems to have recently added new gRPC endpoints (e.g. flight_list) - their definitions are partially added and we can possibly use them in the future

[xoolive]

Hi @cathaypacific8747 will do

[xoolive]

Also tried the text user interface, and it's still working 🥳

[cloudflare-pages]

Deploying flightradar24 with    Cloudflare Pages

Latest commit:	3edf447
Status:	✅  Deploy successful!
Preview URL:	https://8cc46558.flightradar24.pages.dev
Branch Preview URL:	https://new-core.flightradar24.pages.dev

View logs