aiohttp can't send files by POST request correct

[spkps]

I've bumped into multiple different issues when tried to send a file to my REST service. I think it'll be better when I just left some code samples instead of detailed description.

In client part I send two different in size files. I send them using different transports to have the possibility to see errorrs. One of transport is requests and the other is asyncio.

import asyncio
import aiohttp
import requests
import urllib.request

url = 'https://127.0.0.1:8080/load/image'

empty_pixel_url = 'https://upload.wikimedia.org/wikipedia/commons/c/ce/Transparent.gif'
large_image_url = 'https://upload.wikimedia.org/wikipedia/commons/d/df/Comet-SidingSpring-Passing-PlanetMars-On-20141019-ArtistConcept-20140905.jpg'

pixel = {'image': ('pixel.jpg', urllib.request.urlopen(empty_pixel_url))}
large_image = {'image': urllib.request.urlopen(large_image_url)}


@asyncio.coroutine
def post_coroutine(files):
    r = yield from aiohttp.request('post', url, files=files)
    return (yield from r.text())


def post_asyncio(files):
    """
        POST by asyncio
    """
    loop = asyncio.get_event_loop()
    res = loop.run_until_complete(post_coroutine(files))
    print(res)


def post_requests(files):
    """
        POST by requests
    """
    response = requests.post(url, files=files)
    print(response.content)


if __name__ == '__main__':

    print('post by requests\n')
    post_requests(pixel)
    post_requests(large_image)

    print('\n'+'-'*20+'\n')
    print('post by asyncio')
    post_asyncio(pixel)
    post_asyncio(large_image)

The server part has been written with bottlepy.

import time
from pathlib import Path
from bottle import post, run, request

@post('/load/image')
def load_image():
    upload = request.files.get('image')
    fn = upload.filename + str(time.time())
    upload.save(str(Path('/tmp') / fn))
    return dict(filename=fn)

run(host='localhost', port=8080, debug=True)

[kxepal]

For lazy ones aiohttp sends the next data with POST:
b'image=%3Chttp.client.HTTPResponse+object+at+0x7f7e87ae4e10%3E'

[ludovic-gasc]

1. For me, you need to retrieve bytes representation of the content of this object. In your code, you link directly with the HTTPResponse object, it won't work.
2. Avoid to mix sync and async operations: Use aiohttp.request instead of urllib to avoid to block event loop for nothing. It worst case, if you must do sync io with asyncio, please use run_in_executor(): https://docs.python.org/3/library/asyncio-eventloop.html#asyncio.BaseEventLoop.run_in_executor

[spkps]

And one more note for lazy ones: in case with passing of read bytes to files instead of HTTPResponse object, aiohttp sends with POST following:

b'image=pixel.jpg&image=GIF89a%01%00%01%00%80%00%00%00%00%00%FF%FF%FF%21%F9%04%01%00%00%00%00%2C%00%00%00%00%01%00%01%00%00%02%01D%00%3B'

requests in both cases (either HTTPResponce object or raw bytes) sends the same:

b'--4a6bf751a6d844e398a9248af5ca12de\r\nContent-Disposition: form-data; name="image"; filename="pixel.jpg"\r\n\r\nGIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\xff\xff\xff!\xf9\x04\x01\x00\x00\x00\x00,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x01D\x00;\r\n--4a6bf751a6d844e398a9248af5ca12de--\r\n'

I've simplified a little bit the client part. I've removed the example with post_form_coroutine as unnecessary.

[asvetlov]

Well, that's funny and should be fixed. Thanks for report.

[fafhrd91]

@kxepal do you think this issue still true with new multipart support?

[ttezel]

This issue appears invalid now. Reproduced this with:

data = FormData()
data.add_field('foobar', open('Transparent.gif', 'rb'), filename='pixeltez.gif')
r = yield from aiohttp.request('post', url, data=data)
return (yield from r.text())

And it looks like we generate a valid HTTP request body - on the server side I get:

b'--c4a04312312542fc958f61fc4901e41b\r\nCONTENT-TYPE: image/gif\r\nCONTENT-DISPOSITION: form-data; filename="pixeltez.gif"; filename*=utf-8\'\'pixeltez.gif; name=foobar\r\n\r\nGIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\xff\xff\xff!\xf9\x04\x01\x00\x00\x00\x00,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x01D\x00;\r\n--c4a04312312542fc958f61fc4901e41b--\r\n'

[asvetlov]

Ok, thanks.
I'll close the issue, feel free to file new one if the problem is still present.

[lock]

This thread has been automatically locked since there has not been
any recent activity after it was closed. Please open a new issue for
related bugs.

If you feel like there's important points made in this discussion,
please include those exceprts into that new issue.