dynamic page_size for large responses (or configurable)

[bh2smith]

Suggestion

Sometimes the load size returned by RPCs is limited "message":"query returned more than 10000 results. Try with this block range [0x46ADC1, 0x46ADD7].". It would be cool to parse this kind of error and be more dynamic about the query range (since the response actually suggests which block range to use!).

Background

With any of the following Node URLs (alchemy node)

#ethrpc = "https://cloudflare-eth.com/" # Error: JSON error: missing field `data` at line 1 column 136
#ethrpc = "https://rpc.flashbots.net/" # Error: 400 Bad Request:
#ethrpc = "https://rpc.ankr.com/eth" # Unknown Error
# ethrpc = "https://eth-mainnet.g.alchemy.com/v2/<API_KEY>"

when trying to according to the following configuration

[[event]]
name = "nft_mint"
start = 4632001
contract = "*"
topics = [
    "0x0000000000000000000000000000000000000000000000000000000000000000"
]
signature = "event Transfer(address indexed from, address indexed to, uint256 indexed tokenId)"

we get the following error:

Error: JSON error: missing field `data` at line 1 column 117

Caused by:
    missing field `data` at line 1 column 117

I dug a bit and found that the error is raised in ethrpc::http.rs and its not clear how to capture the response:

The query being executed is eth_getLogs

LogFilter { 
    blocks: Range { from: Number(4632001), to: Number(4633000) }, 
    address: Any, 
    topics: [
        Exact(Digest(0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef)),
        Exact(Digest(0x0000000000000000000000000000000000000000000000000000000000000000))
]}

Example

The equivalent curl request is:

curl https://mainnet.infura.io/v3/${INFURA_KEY} \
  -X POST \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","method":"eth_getLogs","params":[{"fromBlock": "0x46adc1","toBlock": "0x46b1a8", "topics":["0xddf252ad1be2c89b69c2b068fc378daa952ba7f163c4a11628f55a4df523b3ef", "0x0000000000000000000000000000000000000000000000000000000000000000"]}],"id":1}'

which responds saying that there are too many. Is it currently possible to configure the block range width?

{"jsonrpc":"2.0","id":1,"error":{"code":-32005,"data":{"from":"0x46ADC1","limit":10000,"to":"0x46ADD7"},"message":"query returned more than 10000 results. Try with this block range [0x46ADC1, 0x46ADD7]."}}%  

[bh2smith]

It seems the project is setup to specify page size as a configuration, but currently uses 1000 by default;

arak/src/config.rs

Lines 123 to 140 in a290291

	mod indexer {
	use {super::Indexer, std::time::Duration};
	pub fn default() -> Indexer {
	Indexer {
	page_size: default_page_size(),
	poll_interval: default_poll_interval(),
	}
	}
	pub fn default_page_size() -> u64 {
	1000
	}
	pub fn default_poll_interval() -> Duration {
	Duration::from_secs_f64(0.1)
	}
	}

[bh2smith]

According to the way the configuration toml is parsed, it looks like the indexer parameters can be set directly in the project toml.

I have tried this:

[indexer]
page_size = 500
poll_interval = 0.1

but it doesn't seem to recognize... Is there something special I need to do to pass "Duration" for poll_interval maybe?