How to Fix 403 Forbidden Errors When Scraping
By Marcus Reiner · 2026-02-27 · 10 min read · Engineering
A 403 isn't always your IP. Here's the diagnostic ladder that finds the real cause in 5 minutes.
Step 1 — reproduce with curl + your proxy
Eliminate your scraper as the variable. `curl -x http://user:pass@proxy:port https://target.com -v` — if curl works and your scraper doesn't, the problem is headers/TLS, not the IP.
Step 2 — check TLS fingerprint
Python `requests` and Node `fetch` have distinctive JA3/JA4 fingerprints. Modern WAFs (Cloudflare, DataDome, Akamai) block them regardless of IP. Swap to curl_cffi, httpx with h2, or a headless browser.
- curl_cffi: `requests.get(url, impersonate='chrome131')`
- Node: undici with custom ClientHello, or use Playwright
- Go: utls library for ClientHello impersonation
Step 3 — upgrade the IP class
If TLS is clean and you still get 403, the IP is the problem. Climb the ladder: datacenter → ISP → residential → mobile. Decodo residential at $2/GB is the cheapest 'works on most things' starting point.
Step 4 — full header set
Send the same headers a real Chrome sends: Accept, Accept-Language, sec-ch-ua, sec-ch-ua-mobile, sec-ch-ua-platform, sec-fetch-dest, sec-fetch-mode, sec-fetch-site, sec-fetch-user, Upgrade-Insecure-Requests. Missing sec-ch-* is a giveaway.
Step 5 — warm the session
Hit the homepage, accept cookies, then navigate. Going straight to a deep URL with zero referer + zero cookies is bot behavior. Reuse the same session for 5-10 requests before rotating.
FAQ
Why does the same URL work in my browser?
Your browser carries cookies, a clean TLS fingerprint, and an IP reputation built over months. Replicate those three and the 403 goes away.