前言
一直以来使用Selenium绕过Cloudflare请求Coinbase做上币监控,但是Selenium非常吃内存,2核4G的机子非常吃力,又在网上找其他的替代方式,发现还可以使用"FlareSolverr"绕过CloudFlare进行网页抓取。
搭建FlareSolverr
FlareSolverr需要单独搭建,使用docker容器可以一键部署
docker run -d \ --name=flaresolverr \ -p 8191:8191 \ -e LOG_LEVEL=info \ --restart unless-stopped \ ghcr.io/flaresolverr/flaresolverr:latest
搭建完成后,在浏览器输入http://ip:8191 (或设置反向代理后使用https://反代域名:8191)
浏览器显示:
{"msg":"FlareSolverr is ready!","version":"v2.2.4","userAgent":"Mozilla/5.0 (X11; Linux x86_64; rv:94.0) Gecko/20100101 Firefox/94.0"}
就表示FlareSolverr已经搭建成功了。
##使用方式
- Example Bash request:
curl -L -X POST 'http://localhost:8191/v1' \
-H 'Content-Type: application/json' \
--data-raw '{
"cmd": "request.get",
"url":"http://www.google.com/",
"maxTimeout": 60000
}'
- Example Python request:
import requests
url = "http://localhost:8191/v1"
headers = {"Content-Type": "application/json"}
data = {
"cmd": "request.get",
"url": "http://www.google.com/",
"maxTimeout": 60000
}
response = requests.post(url, headers=headers, json=data)
print(response.text)