'utf-8' codec can't decode byte 0x?? in position ?: invalid continuation byte

[d-Rickyy-b]

When pastes contain non utf-8 characters, the decoding fails and downloading the paste is being stopped.

Errors logged at:

pastepwn/pastepwn/scraping/pastebin/pastebinscraper.py

Lines 96 to 100 in 1d9b82e

	try:
	response_data = r.get(api_url)
	except Exception as e:
	self.logger.error(e)
	raise e

and:

pastepwn/pastepwn/scraping/pastebin/pastebinscraper.py

Lines 123 to 140 in 1d9b82e

	try:
	body = self._get_paste_content(paste.key)
	except PasteNotReadyException:
	self.logger.debug("Paste '{0}' is not ready for downloading yet. Enqueuing it again.".format(paste.key))
	# Make sure to wait a certain time. If only one element in the queue, this can lead to loops
	self._rate_limit_sleep(last_body_download_time)
	self._tmp_paste_queue.put(paste)
	continue
	except PasteDeletedException:
	# We don't add a sleep here, because this can't lead to loops
	self.logger.info("Paste '{0}' has been deleted before we could download it! Skipping paste.".format(paste.key))
	continue
	except PasteEmptyException:
	self.logger.info("Paste '{0}' is set to None! Skipping paste.".format(paste.key))
	continue
	except Exception as e:
	self.logger.error("An exception occurred while downloading the paste '{0}'. Skipping this paste! Exception is: {1}".format(paste.key, e))
	continue

Example pastes:

2020-02-04 01:17:22,211 - pastepwn.scraping.pastebin.pastebinscraper - ERROR - 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
2020-02-04 01:17:22,213 - pastepwn.scraping.pastebin.pastebinscraper - ERROR - An exception occurred while downloading the paste 'nDPF9r5b'. Skipping this paste! Exception is: 'utf-8' codec can't decode byte 0x8b in position 1: invalid start byte
2020-02-04 01:19:19,130 - pastepwn.scraping.pastebin.pastebinscraper - ERROR - 'utf-8' codec can't decode byte 0xe1 in position 2262: invalid continuation byte
2020-02-04 01:19:19,132 - pastepwn.scraping.pastebin.pastebinscraper - ERROR - An exception occurred while downloading the paste 'aeC9BS25'. Skipping this paste! Exception is: 'utf-8' codec can't decode byte 0xe1 in position 2262: invalid continuation byte
2020-02-04 08:17:52,570 - pastepwn.scraping.pastebin.pastebinscraper - ERROR - 'utf-8' codec can't decode byte 0xe1 in position 2262: invalid continuation byte
2020-02-04 08:17:52,636 - pastepwn.scraping.pastebin.pastebinscraper - ERROR - An exception occurred while downloading the paste '0Cq4CYCH'. Skipping this paste! Exception is: 'utf-8' codec can't decode byte 0xe1 in position 2262: invalid continuation byte

[d-Rickyy-b]

For this bug to be fixed, we need to use the raw bytes. That might break many analyzers.

[issamansur]

Hi, I think that you have problem there:
https://github.com/d-Rickyy-b/pastepwn/blob/1d9b82efa53d948f790b663a54d609150e65b32e/pastepwn/util/request.py#LL37C1-L42C22
You 'except' only TimeoutError add no except other errors.
You need and regex in this part before decoding utf-8.