Scrapy

配置 Scrapy 通过 RaxyProxy 路由所有请求，使用轮换或粘性 IP。

安装

pip install scrapy

settings.py — 全局代理

最简单的方法：在 SCRAPY_PROXY or override HttpProxyMiddleware. Every request in the spider will use a different rotating IP automatically.

settings.py

# ── Proxy ──────────────────────────────────────────────────────────────────
RAXY_USER = 'USERNAME'
RAXY_PASS = 'PASSWORD'
RAXY_HOST = 'proxy.raxyproxy.com'
RAXY_PORT = 823          # HTTP — rotating
# RAXY_PORT = 824        # SOCKS5

PROXY_URL = f'http://{RAXY_USER}:{RAXY_PASS}@{RAXY_HOST}:{RAXY_PORT}'

# Enable Scrapy's built-in proxy middleware
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
}

# Pass proxy URL via environment variable approach (recommended)
import os
os.environ['http_proxy']  = PROXY_URL
os.environ['https_proxy'] = PROXY_URL

# ── Politeness ───────────────────────────────────────────────────────────────
DOWNLOAD_DELAY          = 1
RANDOMIZE_DOWNLOAD_DELAY = True
CONCURRENT_REQUESTS      = 8
AUTOTHROTTLE_ENABLED     = True

自定义中间件

自定义中间件给您完全控制权 — 按请求轮换、注入凭据、处理重试逻辑，以及按爬虫切换代理类型。

middlewares.py

import base64


class RaxyProxyMiddleware:
    """Inject RaxyProxy credentials into every request."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    USER = 'USERNAME'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

settings.py (middleware)

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RaxyProxyMiddleware': 750,
    # Disable the default proxy middleware so they don't conflict
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
}

地理定向

middlewares.py (geo)

import base64


class RaxyGeoMiddleware:
    """Per-spider geo-targeting via spider attribute."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    PASS = 'PASSWORD'

    def _build_user(self, spider):
        base = getattr(spider, 'proxy_user', 'USERNAME')
        geo  = getattr(spider, 'proxy_geo',  None)
        return f'{base}__{geo}' if geo else base

    def process_request(self, request, spider):
        user  = self._build_user(spider)
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{user}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/us_spider.py

import scrapy


class UsSpider(scrapy.Spider):
    name = 'us_spider'

    # Middleware reads these attributes:
    proxy_user = 'USERNAME'
    proxy_geo  = 'cr.us'           # US only — free

    # City-level (2× billing for residential/mobile/datacenter):
    # proxy_geo = 'cr.us;city.newyork'

    start_urls = ['https://httpbin.org/ip']

    def parse(self, response):
        self.logger.info('IP: %s', response.json()['origin'])

粘性会话

基于端口的粘性会话

Ports 10000–20000 端口 10000–20000 各自维持相同的出口 IP。为每个逻辑"会话"分配一个端口，使一个用户的爬取始终从同一个 IP 退出。

middlewares.py (sticky)

import base64


class RaxyStickyMiddleware:
    """Assign a sticky port based on request.meta['session_id']."""

    HOST      = 'proxy.raxyproxy.com'
    USER      = 'USERNAME'
    PASS      = 'PASSWORD'
    BASE_PORT = 10000

    def process_request(self, request, spider):
        session_id = request.meta.get('session_id', 0)
        port  = self.BASE_PORT + (int(session_id) % 10001)   # stay in 10000-20000
        proxy = f'http://{self.HOST}:{port}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/sticky_spider.py

import scrapy


class StickySpider(scrapy.Spider):
    name = 'sticky'

    def start_requests(self):
        for user_id in range(5):
            yield scrapy.Request(
                url='https://httpbin.org/ip',
                meta={'session_id': user_id},
                callback=self.parse,
            )

    def parse(self, response):
        self.logger.info(
            'session %s → %s',
            response.meta['session_id'],
            response.json()['origin'],
        )

按域名轮换

针对不同目标域名使用不同的代理类型 — 公共 API 使用数据中心，受保护的网站使用住宅代理。

middlewares.py (rotating)

import base64


DOMAIN_PROFILES = {
    # datacenter — cheaper, fast
    'api.example.com': {'user': 'USERNAME', 'port': 823},
    # residential geo — protected target
    'shop.example.com': {'user': 'USERNAME__cr.us', 'port': 823},
}

DEFAULT_PROFILE = {'user': 'USERNAME', 'port': 823}


class RaxyDomainMiddleware:
    HOST = 'proxy.raxyproxy.com'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        from urllib.parse import urlparse
        domain  = urlparse(request.url).netloc
        profile = DOMAIN_PROFILES.get(domain, DEFAULT_PROFILE)

        proxy = f'http://{self.HOST}:{profile["port"]}'
        creds = base64.b64encode(
            f'{profile["user"]}:{self.PASS}'.encode()
        ).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

爬取技巧

在代理错误时重试

启用 RETRY_HTTP_CODES，让 Scrapy 在 407（代理认证）和 503 时重试。每次重试都会获得一个新的轮换 IP。

settings.py (retry)

RETRY_ENABLED    = True
RETRY_TIMES      = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 407, 429]

# Respect retry-after headers
AUTOTHROTTLE_ENABLED         = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4

设置	推荐值	原因
`DOWNLOAD_DELAY`	`1–2`	降低对敏感目标的封禁风险
`CONCURRENT_REQUESTS_PER_DOMAIN`	`4`	避免对单个目标造成压力
`COOKIES_ENABLED`	`False`	使用共享 Cookie 的轮换 IP 看起来可疑
`ROBOTSTXT_OBEY`	`True`	默认；仅在目标明确允许时禁用
`DEFAULT_REQUEST_HEADERS`	Set a realistic User-Agent	Scrapy 的默认 UA 有时会被封锁

settings.py (headers)

DEFAULT_REQUEST_HEADERS = {
    'Accept':          'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'User-Agent':      'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0',
}

← 上一页

Puppeteer & Playwright

API 概览