[TODO] Figure out the upstream issue causing ClientPayloadError: Response payload is not completed

[webknjaz]

The currently deployed bots have been experiencing this problem for PATCH and GET requests:

• ClientPayloadError: Response payload is not completed patchback-github-app#41:
  • https://ansible-open-source.sentry.io/issues/4593756606/ / https://ansible-open-source.sentry.io/share/issue/649e905fc9514df48b4d1c16d66ec269/
  • https://ansible-open-source.sentry.io/issues/4590376245/?referrer=github_integration) / https://ansible-open-source.sentry.io/share/issue/02e0b83c1ea642abbd2953511d7c348e/
  • https://ansible-open-source.sentry.io/issues/4620805945/ / https://ansible-open-source.sentry.io/share/issue/e87d38cc9ea448b99182dd48bb92b68f/
• ClientPayloadError: Response payload is not completed chronographer-github-app#37
  • https://ansible-open-source.sentry.io/issues/4630423171/ / https://ansible-open-source.sentry.io/share/issue/65b0266d7524498da89b93be68372c63/

It's been happening since Oct 31 with various levels of intensity. Chronographer only experienced this 3 times all within a 32-second interval, on Nov 16. Patchback has 1-2 errors like this per day.
The bots moved from OpenShift 3 to a new OpenShift 4-based environment on Oct 16, so there is a chance that the CPython runtime environment got updated as well. The python-level deps are pinned via pip-tools. Both have aiohttp==3.6.2, multidict==4.7.6, and yarl==1.5.1. But it's hard to know if the CPython version has changed since I don't have logs and the old OpenShift instance is no longer available. Sentry reports suggest that both bots are running under CPython 3.9.13. The pre-OpenShift configs that used to be used in Heroku have CPython 3.8.6 in them, so maybe old OpenShift was running that.

Also, there are no Sentry reports that would suggest the same problem happening within the old OpenShift (the history goes 90 days back). This probably means that the Fedora container was different and maybe, the CPython version was too.

Upvote & Fund

• We're using Polar.sh so you can upvote and help fund this issue.
• We receive the funding once the issue is completed & confirmed by you.
• Thank you in advance for helping prioritize & fund our backlog.

[webknjaz]

@felixfontein I saw you pinged me somewhere on Matrix, but I lost the notification/room so mentioning you here.

This is essentially what's been happening to Patchback lately.

[webknjaz]

This https://stackoverflow.com/a/70172596/595220 suggests that the problem might only manifest itself if Python is higher, than 3.8.

[webknjaz]

Though, the upstream bug that also shows gidgethub+aiohttp claims that it's present on Python 3.8 too: aio-libs/aiohttp#4581.

[webknjaz]

UPD: with sanitizers/patchback-github-app#41 (comment), we should be getting more details soon.