aiohttp not forwarding cookies with Session requests

[daredevil82]

I'm working on an api gateway service with aiohttp that executes an aggregate operation across a number of internal services and packages them up into a 207 bulk response.

Internally, we use a Django session cookie for the primary means of authentication (planning to move to JWTs in Q2 this year), which means that for the services being hit with this gateway, I need to pass through the cookie from the browser or rest API request.

It seems aiohttp is not doing this at all, nor is it making any cookies available in the Postman Cookies headers available. However, if I hit the service directly with the same cookies, I get the expected response back.

Code right now:

class AggregateView(BaseView):
    async def post(self):
        requests = await self.request.json()
        logger.debug(f'Making [{len(requests)}] service calls at [{datetime.now()}]')
        queries = []
        async with ClientSession(cookie_jar=CookieJar(unsafe=True)) as session:
            for req in requests:
                queries.append(self._retrieve(req, session))
                
            logger.debug(f'Async calls created [{datetime.now()}]')
            results = await asyncio.gather(*queries)
            logger.debug(f'Async calls completed and aggregated [{datetime.now()}]')
            
        return web.json_response(
                status=207,
                data=results
        )

class BaseView(web.View):
    @cache
    async def _retrieve(self, req, session):
        """
        Unpacks a request body and executes the appropriate service call
        :param request: request body as dictionary
        :param session: aiohttp ClientSession
        :return:
        """
        url, path, query_params = await self._unpack_request(req)
        
        logger.info(f'{url}/{path}\t{query_params}')
        logger.info(f'Starting request for url [{url}/{path}] at [{datetime.now()}]')
        logger.info(f'with query params {query_params}')
        
        func = session.get
        params = {
            'url': f'{url}/{path}',
            'params': query_params,
            'headers': {
                'Vary': 'Accept, Origin, Accept, Accept-Encoding, Authorization, Cookie'
            }
        }
        
        async with AsyncBreaker(circuit_breaker, params['url'], func, **params) as resp:
            logger.info(f'received request for url [{url}/{path}] at [{datetime.now()}]')
            logger.info(resp.headers)
            logger.info(resp.cookies)
            return {
                'url': f'{url}/{path}',
                'status': resp.status,
                'response': await resp.json()
            }


def cache(func):
    @functools.wraps(func)
    async def wrapper(*args, **kwargs):
        request = args[1]
        access_logger.info(args[2].cookie_jar._cookies.keys())
        
        requested_service = request.get('service').upper()
        service = SERVICES.get(requested_service)
        if service and service.get('cache'):
            access_logger.info('This response is cacheable.')
            pass
        
        return await func(*args, **kwargs)
        
    return wrapper

dict_keys([]) # output in #cache for cookie jar inspection
This response is cacheable.
Starting request for url [https://service.company.com/api/v1/home-editorial/]
 at [2019-01-11 15:01:03.983993]
with query params None
received request for url [https://service.company.com/api/v1/home-editorial/]
 at [2019-01-11 15:01:04.434696]
Header output: <CIMultiDictProxy('Server': 'nginx/1.11.3', 'Content-Type': 
'application/json', 'Www-Authenticate': 'Token', 'Allow': 'GET, HEAD, 
OPTIONS', 'X-Frame-Options': 'SAMEORIGIN', 'Strict-Transport-Security': 
'max-age=15724800; includeSubDomains; preload', 'Accept-Ranges': 'bytes',
 'Content-Length': '58', 'Accept-Ranges': 'bytes', 'Date': 
'Fri, 11 Jan 2019 15:01:04 GMT', 'Via': '1.1 varnish', 'Connection': 
'keep-alive', 'X-Client-IP': '192.150.73.149', 'X-Served-By': 
'cache-bos8232-BOS', 'X-Cache': 'MISS', 'X-Cache-Hits': '0', 
'X-Timer': 'S1547218864.246313,VS0,VE152', 'Vary': 'Accept, Origin')>

Notes:

• AsyncBreaker uses aiobreaker to use the Circuit Breaker pattern for protecting services from overload
• @cache decorator is currently just wrapping the retrieve method and being used to inspect the attributes of AggregateView and ClientSession. Its going to be more full featured with aiocache once this issue is resolved.

What do I need to do to ensure cookie headers from the incoming request get passed to the subsequent session calls?

[aio-libs-bot]

GitMate.io thinks the contributor most likely able to help you is @asvetlov.

Possibly related issues are #2920 (AIOHttp failing after some requests), #161 (aiohttp can't send files by POST request correct), #206 (SSL issue with aiohttp.request), #139 (Cookies doesn't shared between requests), and #1756 (Memory leak with aiohttp.request).

[asvetlov]

Sorry, your snippet has a lot of incomplete code.
I cannot understand it, and cannot run the code to reproduce the problem.
Can you rework the snippet to provide a minimal reproducible example to demonstrate the issue?

[daredevil82]

I will try, with a github project.

that said, for your own sample projects, how do you forward cookies from incoming request headers?

[asvetlov]

You need to do it explicitly.
Extract cookie params from a web request and pass them to client call.

[cglacet]

@asvetlov Can you point the documentation part for a per-request cookie insertion? The documentation only shows how to add cookies to sessions.

[webknjaz]

The same way: https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientSession.

[cglacet]

Ah ok thanks, I guess you meant this aiohttp.request. I totally missed that "reference" page.