Skip to content

Scrapy download handler that can impersonate browser' TLS signatures or JA3 fingerprints.

License

Notifications You must be signed in to change notification settings

jxlil/scrapy-impersonate

Folders and files

NameName
Last commit message
Last commit date

Latest commit

 

History

49 Commits
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 

Repository files navigation

scrapy-impersonate

version

scrapy-impersonate is a Scrapy download handler. This project integrates curl_cffi to perform HTTP requests, so it can impersonate browsers' TLS signatures or JA3 fingerprints.

Installation

pip install scrapy-impersonate

Activation

Replace the default http and/or https Download Handlers through DOWNLOAD_HANDLERS

DOWNLOAD_HANDLERS = {
    "http": "scrapy_impersonate.ImpersonateDownloadHandler",
    "https": "scrapy_impersonate.ImpersonateDownloadHandler",
}

Also, be sure to install the asyncio-based Twisted reactor:

TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"

Usage

Set the impersonate Request.meta key to download a request using curl_cffi:

import scrapy


class ImpersonateSpider(scrapy.Spider):
    name = "impersonate_spider"
    custom_settings = {
        "DOWNLOAD_HANDLERS": {
            "http": "scrapy_impersonate.ImpersonateDownloadHandler",
            "https": "scrapy_impersonate.ImpersonateDownloadHandler",
        },
        "TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
    }

    def start_requests(self):
        for browser in ["chrome110", "edge99", "safari15_5"]:
            yield scrapy.Request(
                "https://tls.browserleaks.com/json",
                dont_filter=True,
                meta={"impersonate": browser},
            )

    def parse(self, response):
        # ja3_hash: 773906b0efdefa24a7f2b8eb6985bf37
        # ja3_hash: cd08e31494f9531f560d64c695473da9
        # ja3_hash: 2fe1311860bc318fc7f9196556a2a6b9
        yield {"ja3_hash": response.json()["ja3_hash"]}

impersonate-args

You can pass any necessary arguments to curl_cffi through impersonate_args. For example:

yield scrapy.Request(
    "https://tls.browserleaks.com/json",
    dont_filter=True,
    meta={
        "impersonate": browser,
        "impersonate_args": {
            "verify": False,
            "timeout": 10,
        },
    },
)

Supported browsers

The following browsers can be impersonated

Browser Version Build OS Name
Chrome 99 99.0.4844.51 Windows 10 chrome99
Chrome 99 99.0.4844.73 Android 12 chrome99_android
Chrome 100 100.0.4896.75 Windows 10 chrome100
Chrome 101 101.0.4951.67 Windows 10 chrome101
Chrome 104 104.0.5112.81 Windows 10 chrome104
Chrome 107 107.0.5304.107 Windows 10 chrome107
Chrome 110 110.0.5481.177 Windows 10 chrome110
Chrome 116 116.0.5845.180 Windows 10 chrome116
Chrome 119 119.0.6045.199 macOS Sonoma chrome119
Chrome 120 120.0.6099.109 macOS Sonoma chrome120
Chrome 123 123.0.6312.124 macOS Sonoma chrome123
Chrome 124 124.0.6367.60 macOS Sonoma chrome124
Edge 99 99.0.1150.30 Windows 10 edge99
Edge 101 101.0.1210.47 Windows 10 edge101
Safari 15.3 16612.4.9.1.8 MacOS Big Sur safari15_3
Safari 15.5 17613.2.7.1.8 MacOS Monterey safari15_5
Safari 17.0 unclear MacOS Sonoma safari17_0
Safari 17.2 unclear iOS 17.2 safari17_2_ios

Thanks

This project is inspired by the following projects:

  • curl_cffi - Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
  • curl-impersonate - A special build of curl that can impersonate Chrome & Firefox
  • scrapy-playwright - Playwright integration for Scrapy