Scrapy

Configure Scrapy para enrutar todas las solicitudes a través de RaxyProxy con IPs rotativas o sticky.

Instalación

pip install scrapy

settings.py — Proxy global

El enfoque más sencillo: configurar el proxy en SCRAPY_PROXY or override HttpProxyMiddleware. Every request in the spider will use a different rotating IP automatically.

settings.py

# ── Proxy ──────────────────────────────────────────────────────────────────
RAXY_USER = 'USERNAME'
RAXY_PASS = 'PASSWORD'
RAXY_HOST = 'proxy.raxyproxy.com'
RAXY_PORT = 823          # HTTP — rotating
# RAXY_PORT = 824        # SOCKS5

PROXY_URL = f'http://{RAXY_USER}:{RAXY_PASS}@{RAXY_HOST}:{RAXY_PORT}'

# Enable Scrapy's built-in proxy middleware
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
}

# Pass proxy URL via environment variable approach (recommended)
import os
os.environ['http_proxy']  = PROXY_URL
os.environ['https_proxy'] = PROXY_URL

# ── Politeness ───────────────────────────────────────────────────────────────
DOWNLOAD_DELAY          = 1
RANDOMIZE_DOWNLOAD_DELAY = True
CONCURRENT_REQUESTS      = 8
AUTOTHROTTLE_ENABLED     = True

Middleware personalizado

Un middleware personalizado le da control total — rotación por solicitud, inyección de credenciales, lógica de reintento y cambio de tipo de proxy por spider.

middlewares.py

import base64


class RaxyProxyMiddleware:
    """Inject RaxyProxy credentials into every request."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    USER = 'USERNAME'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

settings.py (middleware)

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RaxyProxyMiddleware': 750,
    # Disable the default proxy middleware so they don't conflict
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
}

Geo-Targeting

middlewares.py (geo)

import base64


class RaxyGeoMiddleware:
    """Per-spider geo-targeting via spider attribute."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    PASS = 'PASSWORD'

    def _build_user(self, spider):
        base = getattr(spider, 'proxy_user', 'USERNAME')
        geo  = getattr(spider, 'proxy_geo',  None)
        return f'{base}__{geo}' if geo else base

    def process_request(self, request, spider):
        user  = self._build_user(spider)
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{user}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/us_spider.py

import scrapy


class UsSpider(scrapy.Spider):
    name = 'us_spider'

    # Middleware reads these attributes:
    proxy_user = 'USERNAME'
    proxy_geo  = 'cr.us'           # US only — free

    # City-level (2× billing for residential/mobile/datacenter):
    # proxy_geo = 'cr.us;city.newyork'

    start_urls = ['https://httpbin.org/ip']

    def parse(self, response):
        self.logger.info('IP: %s', response.json()['origin'])

Sesiones sticky

Sesiones sticky basadas en puerto

Ports 10000–20000 Los puertos 10000–20000 mantienen cada uno la misma IP de salida. Asigne un puerto por "sesión" lógica para que el scraping de un usuario siempre salga desde la misma IP.

middlewares.py (sticky)

import base64


class RaxyStickyMiddleware:
    """Assign a sticky port based on request.meta['session_id']."""

    HOST      = 'proxy.raxyproxy.com'
    USER      = 'USERNAME'
    PASS      = 'PASSWORD'
    BASE_PORT = 10000

    def process_request(self, request, spider):
        session_id = request.meta.get('session_id', 0)
        port  = self.BASE_PORT + (int(session_id) % 10001)   # stay in 10000-20000
        proxy = f'http://{self.HOST}:{port}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/sticky_spider.py

import scrapy


class StickySpider(scrapy.Spider):
    name = 'sticky'

    def start_requests(self):
        for user_id in range(5):
            yield scrapy.Request(
                url='https://httpbin.org/ip',
                meta={'session_id': user_id},
                callback=self.parse,
            )

    def parse(self, response):
        self.logger.info(
            'session %s → %s',
            response.meta['session_id'],
            response.json()['origin'],
        )

Rotación por dominio

Diferentes tipos de proxy para diferentes dominios objetivo — datacenter para APIs públicas, residencial para sitios protegidos.

middlewares.py (rotating)

import base64


DOMAIN_PROFILES = {
    # datacenter — cheaper, fast
    'api.example.com': {'user': 'USERNAME', 'port': 823},
    # residential geo — protected target
    'shop.example.com': {'user': 'USERNAME__cr.us', 'port': 823},
}

DEFAULT_PROFILE = {'user': 'USERNAME', 'port': 823}


class RaxyDomainMiddleware:
    HOST = 'proxy.raxyproxy.com'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        from urllib.parse import urlparse
        domain  = urlparse(request.url).netloc
        profile = DOMAIN_PROFILES.get(domain, DEFAULT_PROFILE)

        proxy = f'http://{self.HOST}:{profile["port"]}'
        creds = base64.b64encode(
            f'{profile["user"]}:{self.PASS}'.encode()
        ).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

Consejos de scraping

Reintentar en errores de proxy

Active RETRY_HTTP_CODES para que Scrapy reintente en 407 (auth proxy) y 503. Cada reintento obtiene una IP rotativa nueva.

settings.py (retry)

RETRY_ENABLED    = True
RETRY_TIMES      = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 407, 429]

# Respect retry-after headers
AUTOTHROTTLE_ENABLED         = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4

Configuración	Valor recomendado	Por qué
`DOWNLOAD_DELAY`	`1–2`	Reduce el riesgo de bloqueo en objetivos sensibles
`CONCURRENT_REQUESTS_PER_DOMAIN`	`4`	Evita saturar un único objetivo
`COOKIES_ENABLED`	`False`	Las IPs rotativas con cookies compartidas parecen sospechosas
`ROBOTSTXT_OBEY`	`True`	Por defecto; deshabilitar solo cuando el objetivo lo permita explícitamente
`DEFAULT_REQUEST_HEADERS`	Set a realistic User-Agent	El UA predeterminado de Scrapy a veces es bloqueado

settings.py (headers)

DEFAULT_REQUEST_HEADERS = {
    'Accept':          'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'User-Agent':      'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0',
}

← Anterior

Puppeteer & Playwright

Descripción de la API