PiratProxies

Web scraping

Proxies in Python Requests: working examples

Requests is the standard HTTP library for Python. A proxy is a dictionary passed with the request, and the username and password go inside the proxy address.

The right proxy type for Python Requests

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

The values to enter in Python Requests

Every PiratProxies plan comes down to four values: host, port, username and password. Here they are for Residential; your own username and password are on your plan page.

Hostresi.piratproxies.com
HTTP port8080
SOCKS5 port1080
UsernameUSERNAME
Passwordshown on your plan page

Written as a single address:

http://USERNAME:PASSWORD@resi.piratproxies.com:8080

There is no separate country or session field: both are part of the username. Your plan page has a generator that writes it, and Targeting and sessions describes each piece.

Setting it up in Python Requests

  1. Build the proxy address with your username and password in it.
  2. Pass it for both http and https. The key names the kind of site you are visiting, not the kind of proxy.
  3. Set a timeout, so a slow exit does not hang the script.

Code: Python Requests through a proxy

Python

import requests

proxy = "http://USERNAME:PASSWORD@resi.piratproxies.com:8080"
r = requests.get("https://api.ipify.org", proxies={"http": proxy, "https": proxy}, timeout=30)
print(r.text)

Put your own login from the plan page in place of USERNAME and PASSWORD. The example calls an IP-check service, so what it prints is the address your request came from.

Should Python Requests rotate IPs or keep one?

When Python Requests is fetching lots of unrelated pages, leave the session out of the username and each request gets a fresh IP address. When requests belong together, such as a login, a basket or paginated results that set cookies, put a session in the username so they all share one address.

Here is what a sticky username looks like:

USERNAME-country-us-time-600-session-a1b2c3d4

When Python Requests will not connect

  • Passing only the http key leaves https sites going out directly, from your own IP address. Always set both.
  • For SOCKS5, install requests[socks] and use socks5h:// with port 1080. The h makes the proxy look up the site's name.
  • A requests.Session reuses connections. With a rotating username, the IP address then changes less often than you expect; close the session or send Connection: close to force a new one.
  • Error 407 means the login was refused. Copy the username and password from your plan page once more, and make sure no space crept in and the username was not cut short.
  • An IP in the wrong country nearly always comes from the username, since the country code lives there. Other fixes.

Python Requests and proxies: questions

Can Python Requests use a SOCKS5 proxy?

Yes. Point it at port 1080 with the same username and password.

Which PiratProxies plan suits Python Requests best?

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

Words used in this guide: web scraping, rotating proxy, sticky session, rate limiting, headless browser.

Python Requests belongs to its own developers, and PiratProxies has no connection with them; we name it only to say what this guide covers. Follow the rules of the sites you visit and our acceptable use policy.