RuntimeError: Cannot pause_reading() when closing, triggered via FlowControlStreamReader.feed_data

[zmedico]

Long story short

When trying to read the response to a get request from an https endpoint, I get a RuntimeError: Cannot pause_reading() when closing exception.

Expected behaviour

I expect it to read the response successfully.

Actual behaviour

  File "/usr/lib64/python3.5/site-packages/aiohttp/client_reqrep.py", line 739, in read
    self._content = yield from self.content.read()
  File "/usr/lib64/python3.5/site-packages/aiohttp/streams.py", line 507, in wrapper
    result = yield from func(self, *args, **kw)
  File "/usr/lib64/python3.5/site-packages/aiohttp/streams.py", line 562, in read
    return (yield from super().read(n))
  File "/usr/lib64/python3.5/site-packages/aiohttp/streams.py", line 265, in read
    raise self._exception
  File "/usr/lib64/python3.5/site-packages/aiohttp/parsers.py", line 190, in set_parser
    next(p)
  File "/usr/lib64/python3.5/site-packages/aiohttp/protocol.py", line 313, in __call__
    yield from self.parse_length_payload(out, buf, length)
  File "/usr/lib64/python3.5/site-packages/aiohttp/protocol.py", line 359, in parse_length_payload
    out.feed_data(chunk, len(chunk))
  File "/usr/lib64/python3.5/site-packages/aiohttp/streams.py", line 554, in feed_data
    self._stream.transport.pause_reading()
  File "/usr/lib64/python3.5/asyncio/sslproto.py", line 336, in pause_reading
    self._ssl_protocol._transport.pause_reading()
  File "/usr/lib64/python3.5/asyncio/selector_events.py", line 647, in pause_reading
    raise RuntimeError('Cannot pause_reading() when closing')
RuntimeError: Cannot pause_reading() when closing

Steps to reproduce

The relevant code is something like this, though I've only been able to reproduce it in an application which creates many sessions simultaneously:

async def ssl_fetch(url):
    with aiohttp.ClientSession(
        connector=aiohttp.TCPConnector(
        ssl_context=ssl._create_unverified_context())) as session:

        resp = await session.get(url)
        content = await resp.read()
        await resp.release()
        return content

Your environment

I have observed this bug with aiohttp versions 1.0.1 and 1.0.2 (have not tried any older versions), on Gentoo Linux with Python 3.5.2. I've only been able to reproduce it in an application which creates many sessions simultaneously. Each session has a single request directed at an https endpoint on port 5080 of Linux host running ganeti (https://docs.ganeti.org/ganeti/2.5/man/ganeti-rapi.html).

I was able to workaround the problem by overriding the ClientSession response_class parameter as follows:

class _FlowControlStreamReader(aiohttp.streams.StreamReader):
    def feed_data(self, data, size=0):
        super().feed_data(data)

class _ClientResponse(aiohttp.client_reqrep.ClientResponse):
    # workaround bug in FlowControlStreamReader
    flow_control_class = _FlowControlStreamReader

[ivan-is]

the same error

RuntimeError: Cannot pause_reading() when closing

with this code:

import asyncio
import aiohttp

async def crawler(loop):
    url = 'https://www.powertoolsdirect.com/hand-tools/laser-levels'
    async with aiohttp.ClientSession(loop=loop) as session:
        async with session.get(url) as response:
            print(response.status)

if __name__ == '__main__':
    loop = asyncio.get_event_loop()
    loop.run_until_complete(crawler(loop))
    loop.close() 

environment:

pip3.5 freeze
aiohttp==1.2.0
async-timeout==1.1.0
chardet==2.3.0
yarl==0.8.1

`which python3.5` -V
Python 3.5.2

[IaroslavR]

Confirm this bug on Ubuntu 16.04 with Python 3.6.0b4 and aiohttp 1.2.0
Traceback:

  File "/home/iaroslavr/projects/crawler.py", line 88, in _get_html
    html = await response.text()
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 747, in text
    yield from self.read()
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/client_reqrep.py", line 718, in read
    self._content = yield from self.content.read()
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/streams.py", line 517, in wrapper
    result = yield from func(self, *args, **kw)
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/streams.py", line 582, in read
    return (yield from super().read(n))
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/streams.py", line 264, in read
    raise self._exception
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/parsers.py", line 188, in set_parser
    next(p)
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/protocol.py", line 305, in __call__
    yield from self.parse_chunked_payload(out, buf)
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/protocol.py", line 348, in parse_chunked_payload
    out.feed_data(chunk, len(chunk))
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/protocol.py", line 392, in feed_data
    self.out.feed_data(chunk, len(chunk))
  File "/home/iaroslavr/.virtualenvs/3.6.0b4/lib/python3.6/site-packages/aiohttp/streams.py", line 573, in feed_data
    self._stream.transport.pause_reading()
  File "/usr/local/lib/python3.6.0b4/lib/python3.6/asyncio/sslproto.py", line 344, in pause_reading
    self._ssl_protocol._transport.pause_reading()
  File "/usr/local/lib/python3.6.0b4/lib/python3.6/asyncio/selector_events.py", line 702, in pause_reading
    raise RuntimeError('Cannot pause_reading() when closing')
RuntimeError: Cannot pause_reading() when closing

[fafhrd91]

#1551 should solve this problem. could anyone test it?

[xsandr]

@fafhrd91 I've checked your patch on my staging environment and seems that it fixed the problem! Thanks!

[IaroslavR]

@fafhrd91 Yep, looks like error is gone

[fafhrd91]

fixed by #1551

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.