Proxies for web scraping
Collecting public pages at scale from one IP address gets that address rate-limited or blocked. Spreading requests over many addresses, in the right countries, keeps a crawl running.
Which proxy is best for web scraping?
For most web scraping, rotating residential proxies are the best choice: every request leaves from a different home IP address, so no single address is rate-limited. Use datacenter proxies for sites that accept any visitor, because they cost less, and mobile proxies only for the few sites that challenge residential addresses.
Pick the pool by how strict the site is
Start with the cheapest pool the target accepts. Open data and APIs usually work over datacenter IPs. Sites with bot filtering usually need residential addresses. The few that challenge even those tend to accept mobile carrier addresses.
Rotate for listings, hold for flows
For independent pages, such as search results or product listings, leave the session out of the username and every request gets a new IP. For anything that spans several requests, such as paging through results or a cart, add a session ID so the IP stays the same.
Mind the bill
Traffic is what you pay for. Block images, fonts and media in your scraper, request compressed responses, and cache what does not change.
Stay inside the lines
Collect public data only, respect the law where you and the site are, and do not overload small sites. Our acceptable use policy applies to every plan.
Proxy types that fit
- Residential Lite$0.72 per GB
- Residential$2.64 per GB
- Unlimited Datacenterfrom $0.24 per hour
- IPv6 Datacenter$0.18 per GB
How to set it up
- Buy the smallest Residential Lite plan, one gigabyte, to test with. It is paid in crypto at checkout and delivered by itself.
- Copy the host
lite.piratproxies.com, the port6969, and the username and password from the plan page. - Choose where requests leave from by adding a country code to the username, for example
USERNAME-country-us. - Run the job on your real target. If it works, top up the same plan; if the site blocks it, try the next proxy type in the list above.
Setup guide and code examples · Guides for each tool
Questions about proxies for web scraping
How many proxies do I need for web scraping?
With a rotating plan you do not count proxies. One login gives a new IP address on every connection, so the question becomes how much traffic the job uses, not how many addresses you hold.
Do I need residential proxies for every site?
No. Try datacenter first where the site is open, because it costs far less. Move to residential when you meet blocks or captchas, and to mobile only if those are challenged too.