Scrapy

Configure Scrapy to route all requests through RaxyProxy with rotating or sticky IPs.

Installation

pip install scrapy

settings.py — Global Proxy

The simplest approach: set the proxy in SCRAPY_PROXY or override HttpProxyMiddleware. Every request in the spider will use a different rotating IP automatically.

settings.py

# ── Proxy ──────────────────────────────────────────────────────────────────
RAXY_USER = 'USERNAME'
RAXY_PASS = 'PASSWORD'
RAXY_HOST = 'proxy.raxyproxy.com'
RAXY_PORT = 823          # HTTP — rotating
# RAXY_PORT = 824        # SOCKS5

PROXY_URL = f'http://{RAXY_USER}:{RAXY_PASS}@{RAXY_HOST}:{RAXY_PORT}'

# Enable Scrapy's built-in proxy middleware
DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 750,
}

# Pass proxy URL via environment variable approach (recommended)
import os
os.environ['http_proxy']  = PROXY_URL
os.environ['https_proxy'] = PROXY_URL

# ── Politeness ───────────────────────────────────────────────────────────────
DOWNLOAD_DELAY          = 1
RANDOMIZE_DOWNLOAD_DELAY = True
CONCURRENT_REQUESTS      = 8
AUTOTHROTTLE_ENABLED     = True

Custom Middleware

A custom middleware gives you full control — rotate per request, inject credentials, handle retry logic, and swap proxy type per spider.

middlewares.py

import base64


class RaxyProxyMiddleware:
    """Inject RaxyProxy credentials into every request."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    USER = 'USERNAME'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

settings.py (middleware)

DOWNLOADER_MIDDLEWARES = {
    'myproject.middlewares.RaxyProxyMiddleware': 750,
    # Disable the default proxy middleware so they don't conflict
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': None,
}

Geo-Targeting

middlewares.py (geo)

import base64


class RaxyGeoMiddleware:
    """Per-spider geo-targeting via spider attribute."""

    HOST = 'proxy.raxyproxy.com'
    PORT = 823
    PASS = 'PASSWORD'

    def _build_user(self, spider):
        base = getattr(spider, 'proxy_user', 'USERNAME')
        geo  = getattr(spider, 'proxy_geo',  None)
        return f'{base}__{geo}' if geo else base

    def process_request(self, request, spider):
        user  = self._build_user(spider)
        proxy = f'http://{self.HOST}:{self.PORT}'
        creds = base64.b64encode(f'{user}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/us_spider.py

import scrapy


class UsSpider(scrapy.Spider):
    name = 'us_spider'

    # Middleware reads these attributes:
    proxy_user = 'USERNAME'
    proxy_geo  = 'cr.us'           # US only — free

    # City-level (2× billing for residential/mobile/datacenter):
    # proxy_geo = 'cr.us;city.newyork'

    start_urls = ['https://httpbin.org/ip']

    def parse(self, response):
        self.logger.info('IP: %s', response.json()['origin'])

Sticky Sessions

Port-based sticky sessions

Ports 10000–20000 Ports 10000–20000 each maintain the same exit IP. Assign a port per logical "session" so one user's scrape always exits from the same IP.

middlewares.py (sticky)

import base64


class RaxyStickyMiddleware:
    """Assign a sticky port based on request.meta['session_id']."""

    HOST      = 'proxy.raxyproxy.com'
    USER      = 'USERNAME'
    PASS      = 'PASSWORD'
    BASE_PORT = 10000

    def process_request(self, request, spider):
        session_id = request.meta.get('session_id', 0)
        port  = self.BASE_PORT + (int(session_id) % 10001)   # stay in 10000-20000
        proxy = f'http://{self.HOST}:{port}'
        creds = base64.b64encode(f'{self.USER}:{self.PASS}'.encode()).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

spiders/sticky_spider.py

import scrapy


class StickySpider(scrapy.Spider):
    name = 'sticky'

    def start_requests(self):
        for user_id in range(5):
            yield scrapy.Request(
                url='https://httpbin.org/ip',
                meta={'session_id': user_id},
                callback=self.parse,
            )

    def parse(self, response):
        self.logger.info(
            'session %s → %s',
            response.meta['session_id'],
            response.json()['origin'],
        )

Rotating per Domain

Different proxy types for different target domains — datacenter for public APIs, residential for protected sites.

middlewares.py (rotating)

import base64


DOMAIN_PROFILES = {
    # datacenter — cheaper, fast
    'api.example.com': {'user': 'USERNAME', 'port': 823},
    # residential geo — protected target
    'shop.example.com': {'user': 'USERNAME__cr.us', 'port': 823},
}

DEFAULT_PROFILE = {'user': 'USERNAME', 'port': 823}


class RaxyDomainMiddleware:
    HOST = 'proxy.raxyproxy.com'
    PASS = 'PASSWORD'

    def process_request(self, request, spider):
        from urllib.parse import urlparse
        domain  = urlparse(request.url).netloc
        profile = DOMAIN_PROFILES.get(domain, DEFAULT_PROFILE)

        proxy = f'http://{self.HOST}:{profile["port"]}'
        creds = base64.b64encode(
            f'{profile["user"]}:{self.PASS}'.encode()
        ).decode()

        request.meta['proxy']           = proxy
        request.headers['Proxy-Authorization'] = f'Basic {creds}'

Scraping Tips

Retry on proxy errors

Enable RETRY_HTTP_CODES so Scrapy retries on 407 (proxy auth) and 503. Each retry gets a fresh rotating IP.

settings.py (retry)

RETRY_ENABLED    = True
RETRY_TIMES      = 3
RETRY_HTTP_CODES = [500, 502, 503, 504, 407, 429]

# Respect retry-after headers
AUTOTHROTTLE_ENABLED         = True
AUTOTHROTTLE_TARGET_CONCURRENCY = 4

Setting	Recommended value	Why
`DOWNLOAD_DELAY`	`1–2`	Reduces ban risk on sensitive targets
`CONCURRENT_REQUESTS_PER_DOMAIN`	`4`	Avoids flooding a single target
`COOKIES_ENABLED`	`False`	Rotating IPs with shared cookies look suspicious
`ROBOTSTXT_OBEY`	`True`	Default; disable only when target explicitly allows it
`DEFAULT_REQUEST_HEADERS`	Set a realistic User-Agent	Scrapy's default UA is sometimes blocked

settings.py (headers)

DEFAULT_REQUEST_HEADERS = {
    'Accept':          'text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8',
    'Accept-Language': 'en-US,en;q=0.5',
    'User-Agent':      'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:124.0) Gecko/20100101 Firefox/124.0',
}

← Previous

Puppeteer & Playwright

API Overview