A set of tools to manually ingest Filecoin storage deals and produce a list of retrieval tasks
Make sure you have the latest Node.js LTS version installed.
Remember to install dependencies after cloning this repository locally.
❯ npm install
Generated files are stored in the directory generated
. Make sure to create it before running
the first script.
-
Download the snapshot of StateMarketDeals from Glif: https://marketdeals.s3.amazonaws.com/StateMarketDeals.json.zst
WARNING: The file has more than 3 GB.
-
Build the tool for converting
StateMarketDeals.json
to newline-delimited JSONcargo build --release
-
Run
./target/release/fil-deal-ingester StateMarketDeals.json.zst > generated/StateMarketDeals.ndjson
This will take about 3-5 minutes to complete.
The output is NOT committed to git; you can find it in ./generated/StateMarketDeals.ndjson
.
You can create a smaller file by aborting the jq
command by pressing Ctrl+C and/or truncating the
output file at any line boundary.
-
Run the previous step to build
./generated/StateMarketDeals.ndjsonn
-
Run
node scripts/parse-retrievable-deals.js
This will take several minutes to complete.
The output is NOT committed to git; you can find it in ./generated/retrievable-deals.ndjson
-
Run the previous step to build
./generated/retrievable-deals.ndjson
-
Run
node scripts/build-spark-update-sql.js
The output is NOT committed to git; you can find it in
./generated/update-spark-db.sql
-
Setup port forwarding between your local computer and Postgres instance hosted by Fly.io (docs). Remember to use a different port if you have a local Postgres server for development!
fly proxy 5454:5432 -a spark-db
-
Find spark-db entry in 1Password and get the user and password from the connection string.
-
Run the following command to apply the updates:
psql postgres:https://user:password@localhost:5454/spark -f generated/update-spark-db.sql