The right proxy type for Selenium
For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.
- Residentialfrom $2.64 per GB
- Residential Litefrom $0.60 per GB
- Unlimited Datacenterfrom $0.56 per hour
The values to enter in Selenium
Every PiratProxies plan comes down to four values: host, port, username and password. Here they are for Residential; your own username and password are on your plan page.
| Host | resi.piratproxies.com |
|---|---|
| HTTP port | 8080 |
| SOCKS5 port | 1080 |
| Username | USERNAME |
| Password | shown on your plan page |
Written as a single address:
http://USERNAME:PASSWORD@resi.piratproxies.com:8080There is no separate country or session field: both are part of the username. Your plan page has a generator that writes it, and Targeting and sessions describes each piece.
Setting it up in Selenium
- The simplest route in Python is Selenium Wire, which sits between Selenium and the browser and adds the login.
- Pass the proxy in
seleniumwire_options. - Open an IP-check page to confirm the address.
Code: Selenium through a proxy
Python, with Selenium Wire
from seleniumwire import webdriver # pip install selenium-wire
options = {"proxy": {"http": "http://USERNAME:PASSWORD@resi.piratproxies.com:8080", "https": "http://USERNAME:PASSWORD@resi.piratproxies.com:8080"}}
driver = webdriver.Chrome(seleniumwire_options=options)
driver.get("https://api.ipify.org")
print(driver.find_element("tag name", "body").text)
driver.quit()Put your own login from the plan page in place of USERNAME and PASSWORD. The example calls an IP-check service, so what it prints is the address your request came from.
Should Selenium rotate IPs or keep one?
When Selenium is fetching lots of unrelated pages, leave the session out of the username and each request gets a fresh IP address. When requests belong together, such as a login, a basket or paginated results that set cookies, put a session in the username so they all share one address.
Here is what a sticky username looks like:
USERNAME-country-us-time-600-session-a1b2c3d4When Selenium will not connect
--proxy-server=http://user:pass@host:portdoes not work in Chrome: the login is dropped and you get a password prompt or a 407.- Selenium Wire is no longer developed. A small local forwarder solves this: it listens on your computer without a login and adds the username and password on the way out. See proxies with a login in tools that only take IP and port.
- Error
407means the login was refused. Copy the username and password from your plan page once more, and make sure no space crept in and the username was not cut short. - An IP in the wrong country nearly always comes from the username, since the country code lives there. Other fixes.
Selenium and proxies: questions
Why does Chrome ask for a username and password?
Chrome ignores a login written into the proxy address. Use Selenium Wire, a local forwarder, or a tool such as SeleniumBase that adds the login for you.
Can Selenium use a SOCKS5 proxy?
Only without a login: Selenium cannot send a SOCKS5 username and password. Use the HTTP port, 8080, instead; it handles http and https sites alike.
Which PiratProxies plan suits Selenium best?
For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.
Words used in this guide: web scraping, rotating proxy, sticky session, rate limiting, headless browser.
Selenium belongs to its own developers, and PiratProxies has no connection with them; we name it only to say what this guide covers. Follow the rules of the sites you visit and our acceptable use policy.