Missed txs in blocks

[MaxT-everstake]

We have encountered an issue where nodes with pruning=nothing sometimes skip recording transactions while processing blocks. We observed this on our full nodes, which, despite having identical settings, do not include all transactions. We haven't noticed the absence of transactions in blocks on the mintscan. Has anyone else experienced a similar problem? Is there any way to fix this, perhaps?

[kocubinski]

Can you clarify what you mean a bit by "skip recording transactions"? If a tx was missed in state machine execution nodes would arrive at a different app hash, which doesn't seem to be happening?

[MaxT-everstake]

Sure. For example, I have two full nodes with pruning = "nothing" and indexer = "kv" and when I try to find some tx I see an error 32603 in one of them. The second one gives me the successful answer.
node-1

gaiad q tx 8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6
code: 0
codespace: ""
data: 0A390A372F636F736D6F732E646973747269627574696F6E2E763162657461312E4D7367576974686472617744656C656761746F725265776172640A390A372F636F736D6F732E646973747269627574696F6E2E763162657461312E4D7367576974686472617744656C656761746F72526577617264
events:
......................................................................
  signatures:
  - kHVl2Y8Bw+flQsO/4nG/84mHOYmu6PUvTCCcYuX3NoAriZIYGbNDZpSTXYFI/ERzLVAkf7RzrQThh8/+3lMQNw==
txhash: 8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6

node-2

gaiad q tx 8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6
Error: error in json rpc client, with http response metadata: (Status: 200 OK, Protocol HTTP/1.1). RPC error -32603 - Internal error: tx (8CFC7A4E2A389A9F7D89C03F458A9A25F763ACF40159B55E57586C1E625875D6) not found

[tac0turtle]

this seems like an issue with the indexer, tagging the correct team to help here: @thanethomson

[thanethomson]

Which version of the SDK/Comet is this?

[andynog]

also, do you know if that node-2 had any re-starts around the time that block was created ? like the gaiad might have stopped and re-started ?

[MaxT-everstake]

Both nodes have cosmos_sdk_version: v0.45.16-ics, and I only reboot both nodes one by one during the updates. I've checked block time and reboot time and they mismatch ((