PiratProxies

Web scraping

Proxies in Screaming Frog: working examples

Screaming Frog SEO Spider crawls a site the way a search engine would. It can send the crawl through one proxy, with a username and password.

What a proxy does for Screaming Frog

A proxy lets you crawl a site as a visitor from another country would see it, or keep a large audit from being rate-limited on your office IP address.

The right proxy type for Screaming Frog

Residential proxies in the country you want to audit from. Add a sticky session to the username, so the whole crawl comes from one visitor.

The values to enter in Screaming Frog

Every PiratProxies plan comes down to four values: host, port, username and password. Here they are for Residential; your own username and password are on your plan page.

Hostresi.piratproxies.com
HTTP port8080
SOCKS5 port1080
UsernameUSERNAME
Passwordshown on your plan page

There is no separate country or session field: both are part of the username. Your plan page has a generator that writes it, and Targeting and sessions describes each piece.

Setting it up in Screaming Frog

  1. Open File > Settings > Proxy on Windows or Linux. On a Mac it is under the Screaming Frog SEO Spider menu, in Settings.
  2. Tick Use Proxy Server and enter the host resi.piratproxies.com and the port 8080.
  3. Tick Use Proxy Credentials and enter your username and password.
  4. Click OK and restart the program. The proxy is only used after a restart.

Menus move around between Screaming Frog versions, but the four values you enter never change.

Should Screaming Frog rotate IPs or keep one?

When Screaming Frog is fetching lots of unrelated pages, leave the session out of the username and each request gets a fresh IP address. When requests belong together, such as a login, a basket or paginated results that set cookies, put a session in the username so they all share one address.

Here is what a sticky username looks like:

USERNAME-country-us-time-600-session-a1b2c3d4

When Screaming Frog will not connect

  • The program takes one proxy at a time. For a new IP address on each request, use a rotating username; the rotation then happens on our side.
  • A full crawl of a large site downloads every page and resource. Estimate the size before you start on a gigabyte plan.
  • Error 407 means the login was refused. Copy the username and password from your plan page once more, and make sure no space crept in and the username was not cut short.
  • An IP in the wrong country nearly always comes from the username, since the country code lives there. Other fixes.

Screaming Frog and proxies: questions

Which PiratProxies plan suits Screaming Frog best?

Residential proxies in the country you want to audit from. Add a sticky session to the username, so the whole crawl comes from one visitor.

Words used in this guide: web scraping, rotating proxy, sticky session, rate limiting, headless browser.

Screaming Frog belongs to its own developers, and PiratProxies has no connection with them; we name it only to say what this guide covers. Follow the rules of the sites you visit and our acceptable use policy.