How to Set Up Rotating Proxies in Scrapy 2026
By Marcus Reiner · 2026-05-10 · 10 min read · Engineering
Two patterns: rotating-endpoint and per-request middleware. Code for both, with retry and ban-detection.
Pattern 1 — rotating endpoint (recommended)
Most modern providers expose one URL that returns a fresh IP per connection. In Scrapy:
```python # settings.py DOWNLOADER_MIDDLEWARES = { 'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110, } ``` ```python # spider def start_requests(self): yield Request(url, meta={'proxy': 'http://user:pass@gate.decodo.com:7000'}) ``` Decodo, IPRoyal, Oxylabs, Bright Data all work this way.
Pattern 2 — per-request middleware
For more control, write a middleware that picks an IP per request:
```python class RotatingProxyMiddleware: def __init__(self): self.proxies = open('proxies.txt').read().splitlines() def process_request(self, request, spider): request.meta['proxy'] = random.choice(self.proxies) ```
Retry on ban
```python RETRY_HTTP_CODES = [403, 429, 500, 502, 503, 504] RETRY_TIMES = 3 ``` With a rotating endpoint, each retry gets a fresh IP automatically.
Detect shadow bans
A 200 OK with empty/wrong content is the silent failure mode. Add a content-validator pipeline: if a product page lacks the expected price-selector, raise CloseSpider or retry.
Recommended providers for Scrapy
Decodo ($2/GB) for cost, Bright Data for hardest targets, IPRoyal for budget. Use scrapy-fake-useragent + scrapy-rotating-proxies as community-maintained companion libs.
FAQ
scrapy-rotating-proxies vs provider endpoint?
Use the provider endpoint when you have one — simpler, no IP list to maintain. The library is for cases where you've bought a static IP list.