Scrapy

Configure o Scrapy para rotear todas as requisições pelo RaxyProxy com IPs rotativos ou sticky.

Instalação

pip install scrapy

settings.py — Proxy global

A abordagem mais simples: definir o proxy em SCRAPY_PROXY or override HttpProxyMiddleware. Every request in the spider will use a different rotating IP automatically.

settings.py

# ── Proxy ──────────────────────────────────────────────────────────────────
RAXY_USER = 'USERNAME'
RAXY_PASS = 'PASSWORD'
RAXY_HOST = 'proxy.raxyproxy.com'
RAXY_PORT = 823          # HTTP — rotating
# RAXY_PORT = 824        # SOCKS5

PROXY_URL = f'http://{RAXY_USER}:{RAXY_PASS}@{RAXY_HOST}:{RAXY_PORT}'

# Enable Scrapy's built-in proxy middleware
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
}

# Pass proxy URL via environment variable approach (recommended)
import os
os.environ['http_proxy']  = PROXY_URL
os.environ['https_proxy'] = PROXY_URL

# ── Politeness ───────────────────────────────────────────────────────────────
DOWNLOAD_DELAY          = 1
RANDOMIZE_DOWNLOAD_DELAY = True
CONCURRENT_REQUESTS      = 8
AUTOTHROTTLE_ENABLED     = True

Middleware personalizado

Um middleware personalizado oferece controle total — rotação por requisição, injeção de credenciais, lógica de nova tentativa e troca de tipo de proxy por spider.

middlewares.py

import base64


class RaxyProxyMiddleware:
    """Inject RaxyProxy credentials into every request."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    USER = 'USERNAME'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

settings.py (middleware)

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RaxyProxyMiddleware': 750,
    # Disable the default proxy middleware so they don't conflict
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
}

Geo-Targeting

middlewares.py (geo)

import base64


class RaxyGeoMiddleware:
    """Per-spider geo-targeting via spider attribute."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    PASS = 'PASSWORD'

    def _build_user(self, spider):
        base = getattr(spider, 'proxy_user', 'USERNAME')
        geo  = getattr(spider, 'proxy_geo',  None)
        return f'{base}__{geo}' if geo else base

    def process_request(self, request, spider):
        user  = self._build_user(spider)
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{user}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/us_spider.py

import scrapy


class UsSpider(scrapy.Spider):
    name = 'us_spider'

    # Middleware reads these attributes:
    proxy_user = 'USERNAME'
    proxy_geo  = 'cr.us'           # US only — free

    # City-level (2× billing for residential/mobile/datacenter):
    # proxy_geo = 'cr.us;city.newyork'

    start_urls = ['https://httpbin.org/ip']

    def parse(self, response):
        self.logger.info('IP: %s', response.json()['origin'])

Sessões sticky

Sessões sticky baseadas em porta

Ports 10000–20000 As portas 10000–20000 mantêm cada uma o mesmo IP de saída. Atribua uma porta por "sessão" lógica para que o scraping de um utilizador sempre saia do mesmo IP.

middlewares.py (sticky)

import base64


class RaxyStickyMiddleware:
    """Assign a sticky port based on request.meta['session_id']."""

    HOST      = 'proxy.raxyproxy.com'
    USER      = 'USERNAME'
    PASS      = 'PASSWORD'
    BASE_PORT = 10000

    def process_request(self, request, spider):
        session_id = request.meta.get('session_id', 0)
        port  = self.BASE_PORT + (int(session_id) % 10001)   # stay in 10000-20000
        proxy = f'http://{self.HOST}:{port}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/sticky_spider.py

import scrapy


class StickySpider(scrapy.Spider):
    name = 'sticky'

    def start_requests(self):
        for user_id in range(5):
            yield scrapy.Request(
                url='https://httpbin.org/ip',
                meta={'session_id': user_id},
                callback=self.parse,
            )

    def parse(self, response):
        self.logger.info(
            'session %s → %s',
            response.meta['session_id'],
            response.json()['origin'],
        )

Rotação por domínio

Diferentes tipos de proxy para diferentes domínios alvo — datacenter para APIs públicas, residencial para sites protegidos.

middlewares.py (rotating)

import base64


DOMAIN_PROFILES = {
    # datacenter — cheaper, fast
    'api.example.com': {'user': 'USERNAME', 'port': 823},
    # residential geo — protected target
    'shop.example.com': {'user': 'USERNAME__cr.us', 'port': 823},
}

DEFAULT_PROFILE = {'user': 'USERNAME', 'port': 823}


class RaxyDomainMiddleware:
    HOST = 'proxy.raxyproxy.com'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        from urllib.parse import urlparse
        domain  = urlparse(request.url).netloc
        profile = DOMAIN_PROFILES.get(domain, DEFAULT_PROFILE)

        proxy = f'http://{self.HOST}:{profile["port"]}'
        creds = base64.b64encode(
            f'{profile["user"]}:{self.PASS}'.encode()
        ).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

Dicas de scraping

Tentar novamente em erros de proxy

Ative RETRY_HTTP_CODES para que o Scrapy tente novamente em 407 (auth proxy) e 503. Cada nova tentativa recebe um IP rotativo novo.

settings.py (retry)

RETRY_ENABLED    = True
RETRY_TIMES      = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 407, 429]

# Respect retry-after headers
AUTOTHROTTLE_ENABLED         = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4

Configuração	Valor recomendado	Por quê
`DOWNLOAD_DELAY`	`1–2`	Reduz o risco de bloqueio em alvos sensíveis
`CONCURRENT_REQUESTS_PER_DOMAIN`	`4`	Evita saturar um único alvo
`COOKIES_ENABLED`	`False`	IPs rotativos com cookies compartilhados parecem suspeitos
`ROBOTSTXT_OBEY`	`True`	Padrão; desativar apenas quando o alvo permitir explicitamente
`DEFAULT_REQUEST_HEADERS`	Set a realistic User-Agent	O UA padrão do Scrapy às vezes é bloqueado

settings.py (headers)

DEFAULT_REQUEST_HEADERS = {
    'Accept':          'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'User-Agent':      'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0',
}

← Anterior

Puppeteer & Playwright

Visão geral da API