The ultimate data aggregator using Bitmex's Websocket - FATALITY 💉💀🗡
-
pip3 install git+https://github.com/quan-digital/bitmex-kollector#egg=bitmex_kollector
-
copy, config and run main.py :)
-
data will be stored in settings.DATA_DIR ('data/' by default, change for deploy).
-
status, instrument, candles and indicators will be stored on MongoDB Atlas.
-
run on EC2 eu-west-1 (Ireland) because that's where Bitmex' Websocket is hosted.
-
base Kollector will generate about 140MB of data a day, with more than half going towards 'instrument'. So estimate around 3GB of data a month, or 50GB a year. For bot operation, you can divide that by 60 at least.
"chat", // Trollbox chat - cummulative push
"instrument", // Instrument updates including turnover and bid/ask - continuous push overwrite
"liquidation", // Liquidation orders as they are entered into the book - push refreshed after 20 seconds
"quoteBin1m", // 1-minute quote bins - cummulative push
"tradeBin1m", // 1-minute trade bins - cummulative push
"execution", // Individual executions; can be multiple per order - cummulative push
"margin", // Updates on your current account balance and margin requirements - continuous push overwrite
"position", // Updates on your positions - continuous push overwrite
"transact", // Deposit/Withdrawal updates - cummulative push
For every .csv file, the first two entries are 'DD-MM-YYYY' and 'HH:MM:SS' timestamp components. Remaining columns are listed in order as follows.
Chat data is updated on arrival.
{
"channelID": 0,
"fromBot": false,
"id": 0,
"message": "string",
"user": "string"
}
# "channelID" as follows
{
1 : "English",
2 : "中文",
3 : "Русский",
4 : "한국어",
5 : "日本語",
6 : "Español",
7 : "Français"
}
Liquidation data is updated on arrival.
{
"orderID": "string",
"symbol": "string",
"side": "string",
"price": 0,
"leavesQty": 0
}
Transact data is updated on arrival.
{
"transactID": "string",
"account": 0,
"currency": "string",
"transactType": "string",
"amount": 0,
"fee": 0,
"transactStatus": "string",
"address": "string",
"text": "string"
}
Execution data is updated on arrival.
{
"execID": "string",
"orderID": "string",
"clOrdID": "string",
"account": 0,
"symbol": "string",
"side": "string",
"orderQty": 0,
"price": 0,
"execType": "string",
"ordType": "string",
"commission": 0,
"text": "string"
}
Instrument data is updated every settings.LOOP_INTERVAL seconds (default is 1 second).
{
"symbol": "string",
"state": "string",
"fundingRate": 0,
"indicativeFundingRate": 0,
"prevClosePrice": 0,
"totalVolume": 0,
"volume": 0,
"volume24h": 0,
"totalTurnover": 0,
"turnover": 0,
"turnover24h": 0,
"homeNotional24h": 0,
"foreignNotional24h": 0,
"prevPrice24h": 0,
"vwap": 0,
"highPrice": 0,
"lowPrice": 0,
"lastPrice": 0,
"lastPriceProtected": 0,
"lastTickDirection": "string",
"lastChangePcnt": 0,
"bidPrice": 0,
"midPrice": 0,
"askPrice": 0,
"impactBidPrice": 0,
"impactMidPrice": 0,
"impactAskPrice": 0,
"openInterest": 0,
"openValue": 0,
"markPrice": 0,
"indicativeSettlePrice": 0
}
Trade data is updated every minute.
{
"symbol": "string",
"open": 0,
"high": 0,
"low": 0,
"close": 0,
"trades": 0,
"volume": 0,
"vwap": 0,
"lastSize": 0,
"turnover": 0,
"homeNotional": 0,
"foreignNotional": 0
}
Quote data is updated every minute.
{
"symbol": "string",
"bidSize": 0,
"bidPrice": 0,
"askPrice": 0,
"askSize": 0
}
Margin data is updated on every change.
{
"account": 0,
"currency": "string",
"amount": 0,
"realisedPnl": 0,
"unrealisedPnl": 0,
"indicativeTax": 0,
"unrealisedProfit": 0,
"walletBalance": 0,
"marginBalance": 0,
"marginLeverage": 0,
"marginUsedPcnt": 0,
"availableMargin": 0,
"withdrawableMargin": 0
}
Position data is updated on every change.
{
"account": 0,
"symbol": "string",
"commission": 0,
"leverage": 0,
"crossMargin": true,
"rebalancedPnl": 0,
"openOrderBuyQty": 0,
"openOrderBuyCost": 0,
"openOrderSellQty": 0,
"openOrderSellCost": 0,
"execBuyQty": 0,
"execBuyCost": 0,
"execSellQty": 0,
"execSellCost": 0,
"currentQty": 0,
"currentCost": 0,
"isOpen": true,
"markPrice": 0,
"markValue": 0,
"homeNotional": 0,
"foreignNotional": 0,
"posState": "string",
"realisedPnl": 0,
"unrealisedPnl": 0,
"avgCostPrice": 0,
"avgEntryPrice": 0,
"breakEvenPrice": 0,
"liquidationPrice": 0,
"bankruptPrice": 0
}
Subscription: Consumes one token from your request limiter.
Connection: 20 per hour, on a separate limiter.
# Responses
{"status":429,"error":"Rate limit exceeded, retry in 1 seconds.","meta":{"retryAfter":1},"request":{"op":"subscribe","args":"orderBook"}}
HTTP/1.1 429 Too Many Requests
Cache-Control: no-store, no-cache, must-revalidate, max-age=0
Pragma: no-cache
X-RateLimit-Limit: 20
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1506983924
Retry-After: 29
Content-Type: application/json
Content-Length: 55
Date: Mon, 02 Oct 2017 21:43:49 GMT
Connection: keep-alive
{"error":"Rate limit exceeded, retry in 29 seconds."}
"announcement", // Site announcements
"chat", // Trollbox chat
"connected", // Statistics of connected users/bots
"funding", // Updates of swap funding rates. Sent every funding interval (usually 8hrs)
"instrument", // Instrument updates including turnover and bid/ask
"insurance", // Daily Insurance Fund updates
"liquidation", // Liquidation orders as they are entered into the book
"orderBookL2_25", // Top 25 levels of level 2 order book
"orderBookL2", // Full level 2 order book
"orderBook10", // Top 10 levels using traditional full book push
"publicNotifications", // System-wide notifications (used for short-lived messages)
"quote", // Top level of the book
"quoteBin1m", // 1-minute quote bins
"quoteBin5m", // 5-minute quote bins
"quoteBin1h", // 1-hour quote bins
"quoteBin1d", // 1-day quote bins
"settlement", // Settlements
"trade", // Live trades
"tradeBin1m", // 1-minute trade bins
"tradeBin5m", // 5-minute trade bins
"tradeBin1h", // 1-hour trade bins
"tradeBin1d" // 1-day trade bins
"affiliate", // Affiliate status, such as total referred users & payout % - push refreshed
"execution", // Individual executions; can be multiple per order
"order", // Live updates on your orders - continuous push overwrite
"margin", // Updates on your current account balance and margin requirements
"position", // Updates on your positions
"privateNotifications", // Individual notifications - currently not used
"transact", // Deposit/Withdrawal updates
"wallet" // Bitcoin address balance data, including total deposits & withdrawals - continuous push overwrite
Left out due to low update frequency/importance.
"announcement", // Site announcements - low frequency & easily fetchable via rest
"connected", // Statistics of connected users/bots - apparently discontinued
"insurance", // Daily Insurance Fund updates - liquidation already gets it, continuous push overwrite
"orders" // Live updates on your orders - our bots already deals with orders via POST orders anyway, and generic order data is already fetched through the 'position' topic, continuous push overwrite