PiratProxies

Web scraping

How to use proxies with undetected-chromedriver

undetected-chromedriver is a patched ChromeDriver for Python. Like Chrome itself, it takes a proxy address but not a username and password.

Which proxy to use

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

What to enter

A PiratProxies plan gives you four things: a host, a port, a username and a password. These are the values for Residential; the username and password are on your plan page.

Hostresi.piratproxies.com
HTTP port8080
SOCKS5 port1080
UsernameUSERNAME
Passwordfrom your plan page

The country and the session are chosen in the username, not in a separate setting. The generator on your plan page writes the username for you, and Targeting and sessions explains each part.

Steps

  1. Start a local forwarder that listens without a login and passes traffic on to the proxy with your login.
  2. Point the browser at the forwarder with --proxy-server.

Example

Start a forwarder (gost)

gost -L=:8899 -F=http://USERNAME:PASSWORD@resi.piratproxies.com:8080

Python

import undetected_chromedriver as uc

options = uc.ChromeOptions()
options.add_argument("--proxy-server=http://127.0.0.1:8899")
driver = uc.Chrome(options=options)
driver.get("https://api.ipify.org")
print(driver.find_element("tag name", "body").text)
driver.quit()

Replace USERNAME and PASSWORD with the login from your plan page. The example asks an IP-check address, so the output is the address your request left from.

Rotating or sticky

For crawling many pages with undetected-chromedriver, leave the username without a session so every request leaves from a new IP address. For anything with steps that belong together, such as a login, a cart or a paginated search that sets cookies, add a session to the username so those requests share one address.

A sticky username looks like this:

USERNAME-country-us-time-600-session-a1b2c3d4

If it does not work

  • A small local forwarder solves this: it listens on your computer without a login and adds the username and password on the way out. See proxies with a login in tools that only take IP and port.
  • Run one forwarder per IP address you want. Each can carry a different session ID in its username.
  • A 407 error means the username or password was not accepted. Copy both again from your plan page, and check that nothing added a space or cut the username short.
  • If the IP address is in the wrong country, look at the username: the country code is part of it. More fixes.

Questions

Does undetected-chromedriver work with SOCKS5 proxies?

Not with a username and password: undetected-chromedriver can use SOCKS5 only without a login. Use the HTTP port, 8080, which works for both http and https sites.

Which PiratProxies proxy should I use with undetected-chromedriver?

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

undetected-chromedriver is a product of its own makers. PiratProxies is not affiliated with it; the name is used here only to say what this guide is about. Use it within the rules of the sites you work with and our acceptable use policy.