skip callbacks when a hoster is unreachable & cleanup/refactoring #22
Conversation
| repos = run_block(block_data) | ||
| except (MaxRetryError, ConnectionError, Timeout, TooManyRedirects): | ||
| logger.exception("hosting service not reachable - no indexer callback issued") | ||
| return | ||
| _hoster_session_request( | ||
| "PUT", session, url=block_data[BLOCK_KEY_CALLBACK_URL], json=repos | ||
| ) |
There was a problem hiding this comment.
this is the main part that fixes the potential issue - we simply dont issue the callback request if we encounter these specific exceptions.
All other exceptions are handled like previously, meaning they will only cause the chunk where they occur in to be empty - this one instead ignores any crawled content (if any) and just drops the block without a callback, and asks for the next block (if we're in the automated workflow like in docker-compose).
There was a problem hiding this comment.
im not sure anymore if we can deal with it like that.
since we only clean up dead blocks on add_blocks (when the crawler sends the "answer"), we would only ever hand out more blocks, but never clean up these dead blocks.
also, right now, in the indexer we are scheduling blocks like "hoster with the oldest run timestamp first", meaning that crawling would be stuck on this hoster the indexer never get answers for. (and even if we change the way we schedule, this hoster would only pile up more and more dead blocks, until redis uses up all the ram and crashes)
maybe we should think about a proper communication protocol first, something that wraps the repo list we return in some kind of state, so that the indexer has a way to, for example, pause a run on a hoster for a day on connection errors, or end the run completely without making an export - something like that?
crawlers/lib/platforms/gitea.py
Outdated
| except Exception as e: | ||
| logger.exception(f"(skipping block chunk) gitea crawler crashed") | ||
| return False, [], state # nr.2 - we skip rest of this block, hope we get it next time | ||
| return False, [], state, e # nr.2 - we skip rest of this block, hope we get it next time |
There was a problem hiding this comment.
First except with specific exceptions are the ones we want to re-raise as they mean we have a complete failure for the block - we could try to salvage these requests more but I think we should start like this.
the other side, all other exceptions are like I said handled like before, we catch them and continue (but now we also yield any caught exceptions that occured).
No description provided.