PiratProxies

Web scraping

Proxies in Octoparse: working examples

Octoparse is a point-and-click scraper. It accepts your own proxies for tasks that run on your computer, but only as an IP address and port, without a username and password.

What a proxy does for Octoparse

Because Octoparse has nowhere to type a login, a proxy that needs one cannot be entered directly. A small local forwarder bridges the gap.

The right proxy type for Octoparse

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

The values to enter in Octoparse

Every PiratProxies plan comes down to four values: host, port, username and password. Here they are for Residential; your own username and password are on your plan page.

Hostresi.piratproxies.com
HTTP port8080
SOCKS5 port1080
UsernameUSERNAME
Passwordshown on your plan page

There is no separate country or session field: both are part of the username. Your plan page has a generator that writes it, and Targeting and sessions describes each piece.

Setting it up in Octoparse

  1. Start a local forwarder that listens without a login and passes traffic to your proxy with the login added. With gost: gost -L=:8899 -F=http://USERNAME:PASSWORD@resi.piratproxies.com:8080.
  2. In Octoparse, open the task's settings and find the anti-blocking section.
  3. Choose Use my own proxies, click Configure and add 127.0.0.1:8899.
  4. Run the task locally.

Menus move around between Octoparse versions, but the four values you enter never change.

Should Octoparse rotate IPs or keep one?

When Octoparse is fetching lots of unrelated pages, leave the session out of the username and each request gets a fresh IP address. When requests belong together, such as a login, a basket or paginated results that set cookies, put a session in the username so they all share one address.

Here is what a sticky username looks like:

USERNAME-country-us-time-600-session-a1b2c3d4

When Octoparse will not connect

  • A small local forwarder solves this: it listens on your computer without a login and adds the username and password on the way out. See proxies with a login in tools that only take IP and port.
  • Your own proxies apply to local runs. Tasks run in Octoparse's cloud use Octoparse's proxies instead.
  • Error 407 means the login was refused. Copy the username and password from your plan page once more, and make sure no space crept in and the username was not cut short.
  • An IP in the wrong country nearly always comes from the username, since the country code lives there. Other fixes.

Octoparse and proxies: questions

Can I enter a proxy username and password in Octoparse?

No. Its own-proxy setting takes an IP address and a port only. Run a local forwarder that adds the login, and give Octoparse the forwarder's address.

Which PiratProxies plan suits Octoparse best?

For most scraping, Residential Lite is the cheapest way to get home IP addresses in the right country. Use Residential when you need a particular state or city, and Unlimited Datacenter for targets in the United States that do not block data centre addresses.

Words used in this guide: web scraping, rotating proxy, sticky session, rate limiting, headless browser.

Octoparse belongs to its own developers, and PiratProxies has no connection with them; we name it only to say what this guide covers. Follow the rules of the sites you visit and our acceptable use policy.