scrapy-impersonate is a Scrapy download handler. This project integrates curl_cffi to perform HTTP requests, so it can impersonate browsers' TLS signatures or JA3 fingerprints.
pip install scrapy-impersonate
To use this package, replace the default http and https Download Handlers by updating the DOWNLOAD_HANDLERS setting:
DOWNLOAD_HANDLERS = {
"http": "scrapy_impersonate.ImpersonateDownloadHandler",
"https": "scrapy_impersonate.ImpersonateDownloadHandler",
}By setting USER_AGENT = None, curl_cffi will automatically choose the appropriate User-Agent based on the impersonated browser:
USER_AGENT = NoneAlso, be sure to install the asyncio-based Twisted reactor for proper asynchronous execution:
TWISTED_REACTOR = "twisted.internet.asyncioreactor.AsyncioSelectorReactor"Set the impersonate Request.meta key to download a request using curl_cffi:
import scrapy
class ImpersonateSpider(scrapy.Spider):
name = "impersonate_spider"
custom_settings = {
"TWISTED_REACTOR": "twisted.internet.asyncioreactor.AsyncioSelectorReactor",
"USER_AGENT": None,
"DOWNLOAD_HANDLERS": {
"http": "scrapy_impersonate.ImpersonateDownloadHandler",
"https": "scrapy_impersonate.ImpersonateDownloadHandler",
},
"DOWNLOADER_MIDDLEWARES": {
"scrapy_impersonate.RandomBrowserMiddleware": 1000,
},
}
def start_requests(self):
for _ in range(5):
yield scrapy.Request(
"https://tls.browserleaks.com/json",
dont_filter=True,
)
def parse(self, response):
# ja3_hash: 98cc085d47985d3cca9ec1415bbbf0d1 (chrome133a)
# ja3_hash: 2d692a4485ca2f5f2b10ecb2d2909ad3 (firefox133)
# ja3_hash: c11ab92a9db8107e2a0b0486f35b80b9 (chrome124)
# ja3_hash: 773906b0efdefa24a7f2b8eb6985bf37 (safari15_5)
# ja3_hash: cd08e31494f9531f560d64c695473da9 (chrome99_android)
yield {"ja3_hash": response.json()["ja3_hash"]}You can pass any necessary arguments to curl_cffi through impersonate_args. For example:
yield scrapy.Request(
"https://tls.browserleaks.com/json",
dont_filter=True,
meta={
"impersonate": browser,
"impersonate_args": {
"verify": False,
"timeout": 10,
},
},
)The following browsers can be impersonated
| Browser | Version | Build | OS | Name |
|---|---|---|---|---|
| 99 | 99.0.4844.51 | Windows 10 | chrome99 |
|
| 99 | 99.0.4844.73 | Android 12 | chrome99_android |
|
| 100 | 100.0.4896.75 | Windows 10 | chrome100 |
|
| 101 | 101.0.4951.67 | Windows 10 | chrome101 |
|
| 104 | 104.0.5112.81 | Windows 10 | chrome104 |
|
| 107 | 107.0.5304.107 | Windows 10 | chrome107 |
|
| 110 | 110.0.5481.177 | Windows 10 | chrome110 |
|
| 116 | 116.0.5845.180 | Windows 10 | chrome116 |
|
| 119 | 119.0.6045.199 | macOS Sonoma | chrome119 |
|
| 120 | 120.0.6099.109 | macOS Sonoma | chrome120 |
|
| 123 | 123.0.6312.124 | macOS Sonoma | chrome123 |
|
| 124 | 124.0.6367.60 | macOS Sonoma | chrome124 |
|
| 131 | 131.0.6778.86 | macOS Sonoma | chrome131 |
|
| 131 | 131.0.6778.81 | Android 14 | chrome131_android |
|
| 133 | 133.0.6943.55 | macOS Sequoia | chrome133a |
|
| 99 | 99.0.1150.30 | Windows 10 | edge99 |
|
| 101 | 101.0.1210.47 | Windows 10 | edge101 |
|
| 15.3 | 16612.4.9.1.8 | MacOS Big Sur | safari15_3 |
|
| 15.5 | 17613.2.7.1.8 | MacOS Monterey | safari15_5 |
|
| 17.0 | unclear | MacOS Sonoma | safari17_0 |
|
| 17.2 | unclear | iOS 17.2 | safari17_2_ios |
|
| 18.0 | unclear | MacOS Sequoia | safari18_0 |
|
| 18.0 | unclear | iOS 18.0 | safari18_0_ios |
|
| 133.0 | 133.0.3 | macOS Sonoma | firefox133 |
|
| 135.0 | 135.0.1 | macOS Sonoma | firefox135 |
This project is inspired by the following projects:
- curl_cffi - Python binding for curl-impersonate via cffi. A http client that can impersonate browser tls/ja3/http2 fingerprints.
- curl-impersonate - A special build of curl that can impersonate Chrome & Firefox
- scrapy-playwright - Playwright integration for Scrapy