Scrapy

Configurez Scrapy pour acheminer toutes les requêtes via RaxyProxy avec des IPs rotatives ou sticky.

Installation

pip install scrapy

settings.py — Proxy global

L'approche la plus simple : définir le proxy dans SCRAPY_PROXY or override HttpProxyMiddleware. Every request in the spider will use a different rotating IP automatically.

settings.py

# ── Proxy ──────────────────────────────────────────────────────────────────
RAXY_USER = 'USERNAME'
RAXY_PASS = 'PASSWORD'
RAXY_HOST = 'proxy.raxyproxy.com'
RAXY_PORT = 823          # HTTP — rotating
# RAXY_PORT = 824        # SOCKS5

PROXY_URL = f'http://{RAXY_USER}:{RAXY_PASS}@{RAXY_HOST}:{RAXY_PORT}'

# Enable Scrapy's built-in proxy middleware
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
}

# Pass proxy URL via environment variable approach (recommended)
import os
os.environ['http_proxy']  = PROXY_URL
os.environ['https_proxy'] = PROXY_URL

# ── Politeness ───────────────────────────────────────────────────────────────
DOWNLOAD_DELAY          = 1
RANDOMIZE_DOWNLOAD_DELAY = True
CONCURRENT_REQUESTS      = 8
AUTOTHROTTLE_ENABLED     = True

Middleware personnalisé

Un middleware personnalisé vous donne un contrôle total — rotation par requête, injection de credentials, gestion de la logique de réessai et changement de type de proxy par spider.

middlewares.py

import base64


class RaxyProxyMiddleware:
    """Inject RaxyProxy credentials into every request."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    USER = 'USERNAME'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

settings.py (middleware)

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RaxyProxyMiddleware': 750,
    # Disable the default proxy middleware so they don't conflict
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
}

Geo-Targeting

middlewares.py (geo)

import base64


class RaxyGeoMiddleware:
    """Per-spider geo-targeting via spider attribute."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    PASS = 'PASSWORD'

    def _build_user(self, spider):
        base = getattr(spider, 'proxy_user', 'USERNAME')
        geo  = getattr(spider, 'proxy_geo',  None)
        return f'{base}__{geo}' if geo else base

    def process_request(self, request, spider):
        user  = self._build_user(spider)
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{user}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/us_spider.py

import scrapy


class UsSpider(scrapy.Spider):
    name = 'us_spider'

    # Middleware reads these attributes:
    proxy_user = 'USERNAME'
    proxy_geo  = 'cr.us'           # US only — free

    # City-level (2× billing for residential/mobile/datacenter):
    # proxy_geo = 'cr.us;city.newyork'

    start_urls = ['https://httpbin.org/ip']

    def parse(self, response):
        self.logger.info('IP: %s', response.json()['origin'])

Sessions sticky

Sessions sticky basées sur le port

Ports 10000–20000 Les ports 10000–20000 maintiennent chacun la même IP de sortie. Attribuez un port par "session" logique pour que le scraping d'un utilisateur sorte toujours de la même IP.

middlewares.py (sticky)

import base64


class RaxyStickyMiddleware:
    """Assign a sticky port based on request.meta['session_id']."""

    HOST      = 'proxy.raxyproxy.com'
    USER      = 'USERNAME'
    PASS      = 'PASSWORD'
    BASE_PORT = 10000

    def process_request(self, request, spider):
        session_id = request.meta.get('session_id', 0)
        port  = self.BASE_PORT + (int(session_id) % 10001)   # stay in 10000-20000
        proxy = f'http://{self.HOST}:{port}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/sticky_spider.py

import scrapy


class StickySpider(scrapy.Spider):
    name = 'sticky'

    def start_requests(self):
        for user_id in range(5):
            yield scrapy.Request(
                url='https://httpbin.org/ip',
                meta={'session_id': user_id},
                callback=self.parse,
            )

    def parse(self, response):
        self.logger.info(
            'session %s → %s',
            response.meta['session_id'],
            response.json()['origin'],
        )

Rotation par domaine

Différents types de proxy pour différents domaines cibles — datacenter pour les API publiques, résidentiel pour les sites protégés.

middlewares.py (rotating)

import base64


DOMAIN_PROFILES = {
    # datacenter — cheaper, fast
    'api.example.com': {'user': 'USERNAME', 'port': 823},
    # residential geo — protected target
    'shop.example.com': {'user': 'USERNAME__cr.us', 'port': 823},
}

DEFAULT_PROFILE = {'user': 'USERNAME', 'port': 823}


class RaxyDomainMiddleware:
    HOST = 'proxy.raxyproxy.com'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        from urllib.parse import urlparse
        domain  = urlparse(request.url).netloc
        profile = DOMAIN_PROFILES.get(domain, DEFAULT_PROFILE)

        proxy = f'http://{self.HOST}:{profile["port"]}'
        creds = base64.b64encode(
            f'{profile["user"]}:{self.PASS}'.encode()
        ).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

Conseils de scraping

Réessayer sur les erreurs proxy

Activez RETRY_HTTP_CODES pour que Scrapy réessaie sur 407 (auth proxy) et 503. Chaque réessai obtient une nouvelle IP rotative.

settings.py (retry)

RETRY_ENABLED    = True
RETRY_TIMES      = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 407, 429]

# Respect retry-after headers
AUTOTHROTTLE_ENABLED         = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4

Paramètre	Valeur recommandée	Pourquoi
`DOWNLOAD_DELAY`	`1–2`	Réduit le risque de blocage sur les cibles sensibles
`CONCURRENT_REQUESTS_PER_DOMAIN`	`4`	Évite de saturer une seule cible
`COOKIES_ENABLED`	`False`	Les IPs rotatives avec des cookies partagés semblent suspectes
`ROBOTSTXT_OBEY`	`True`	Par défaut ; désactiver uniquement quand la cible l'autorise explicitement
`DEFAULT_REQUEST_HEADERS`	Set a realistic User-Agent	L'UA par défaut de Scrapy est parfois bloqué

settings.py (headers)

DEFAULT_REQUEST_HEADERS = {
    'Accept':          'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'User-Agent':      'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0',
}

← Précédent

Puppeteer & Playwright

Vue d'ensemble de l'API