Overview

BotProxy Rotating HTTP Proxy allows you to configure a single proxy in your application and automatically distribute your requests through our servers located in different geographical regions around the world. We have locations in the US, Canada, Europe, Asia, and Australia. Each outgoing proxy changes its IP address every 24 hours, giving you access to up to 3,000 fresh IP addresses each month.

BotProxy is a low-level solution designed for integration with existing software. If you need a higher-level API, consider using our browser-based scraping API service, FetchBytes.com, where you can fully render and interact with websites through a simple API.

If you are a website owner and want to block access to your website through our service please contact support, your website will be blacklisted on our service immediately. Access to Facebook, Instagram and Watsapp as well as to some other websites is blocked. We encourage you to take a free trial and test if our service works for you before making a payment.

Quick Start

Select your programming language to see how to integrate BotProxy into your application. Each request should be authenticated by login and password or by whitelisting origin IP address (see Authentication section below).

curl --proxy x.botproxy.net:8080 --proxy-user user-key:key-password "https://httpbin.org/ip"
              

package example

import (
"encoding/base64"
"fmt"
"io/ioutil"
"net/http"
"net/url"
)

func basicAuth(username, password string) string {
  auth := username + ":" + password
  return base64.StdEncoding.EncodeToString([]byte(auth))
}

func main() {
  tr := &http.Transport{
  Proxy: func(req *http.Request) (*url.URL, error) {
    return url.Parse("http://x.botproxy.net:8080") // For secure proxy: url.Parse("https://x.botproxy.net:8443")
    },
  }

  client := &http.Client{Transport: tr}
  req, _ := http.NewRequest("GET", "https://httpbin.org/ip", nil)
  req.Header.Add("Proxy-Authorization", "Basic "+basicAuth("pxu10000-0", "ProxyUser_password"))

  res, err := client.Do(req)
  if err != nil {
    fmt.Println("HTTP error: ", err)
  }

  page, _ := ioutil.ReadAll(res.Body)
  res.Body.Close()
  fmt.Println(page)
}
              

#!/usr/bin/env node
require('request-promise')({
url: 'https://httpbin.org/ip',
proxy: 'http://user-key:[email protected]:8080'
}).then(function(data){ console.log(data); }, function(err){ console.error(err); });
              

package example;

import org.apache.http.HttpHost;
import org.apache.http.client.fluent.*;

public class Example {
  public static void main(String[] args) throws Exception {
    HttpHost proxy = new HttpHost("x.botproxy.net", 8080);
    String res = Executor.newInstance()
      .auth(proxy, "user-key", "key-password")
      .execute(Request.Get("https://httpbin.org/ip").viaProxy(proxy))
      .returnContent().asString();
    System.out.println(res);
  }
}
              

using System;
using System.Net;

class Example
{
  static void Main()
  {
    var client = new WebClient();
    client.Proxy = new WebProxy("x.botproxy.net:8080");
    client.Proxy.Credentials = new NetworkCredential("user-key", "key-password");
    Console.WriteLine(client.DownloadString("https://httpbin.org/ip"));
  }
}
              

Imports System.Net

Module Example
Sub Main()
Dim Client As New WebClient
Client.Proxy = New WebProxy("http://x.botproxy.net:8080")
Client.Proxy.Credentials = New NetworkCredential("user-key", "key-password")
Console.WriteLine(Client.DownloadString("https://httpbin.org/ip"))
End Sub
End Module
              

<?php
$curl = curl_init('https://httpbin.org/ip');
curl_setopt($curl, CURLOPT_PROXY, 'http://x.botproxy.net:8080');
curl_setopt($curl, CURLOPT_PROXYUSERPWD, 'user-key:key-password');
curl_exec($curl);
?>
              

#!/usr/bin/env python
import urllib.request
opener = urllib.request.build_opener(
  urllib.request.ProxyHandler(
    {'http': 'http://user-key:[email protected]:8080',
     'https': 'http://user-key:[email protected]:8080'}))
print(opener.open('https://httpbin.org/ip').read())
              

import requests

res = requests.get(
  'http://httpbin.org/ip',
  proxies={
    'http': 'http://user-key:[email protected]:8080',
    'https': 'http://user-key:[email protected]:8080'
  },
)
print(res.text)
              

#!/usr/bin/ruby

require 'uri'
require 'net/http'

uri = URI.parse('https://httpbin.org/ip')
proxy = Net::HTTP::Proxy('x.botproxy.net', 8080, 'user-key', 'key-password')

req = Net::HTTP::Get.new(uri.path)

result = proxy.start(uri.host,uri.port) do |http|
http.request(req)
end

puts result.body
                

#!/usr/bin/perl
use LWP::UserAgent;
my $agent = LWP::UserAgent->new();
$agent->proxy(['http', 'https'], "http://user-key:key-password\@x.botproxy.net:8080");
print $agent->get('https://httpbin.org/ip')->content();
                

Authentication

Our proxies require authentication with each request. You need to provide proxy user login and password or whitelist your server IP addresses. These are not the login and password you used to sign up on our service. Proxy user login starts with pxu and looks like pxu1000-0.

You can find your proxy user credentials on your account page. Click on edit user button to access proxy user settings form where you can change password or whitelist IPs.

IP Rotation

Each request you make goes through our superproxy you connect to (x.botproxy.net or x1.botproxy.net) and one of the outgoing peer nodes. Superproxy selects outgoing peer randomly from the pool available to your account. Selected peer is stored in session and can be reused in next requests. Once the session expires (default is 60 seconds) a new session is started and a new IP address is selected. In addition to that all peers change their IPs every 24 hours or more often.

You can change maximum session age in proxy user details form to adjust for longer or shorter sessions. You can start a new session with each request to change IP every time, but we do not recommend doing so. Read more about session management in detail below.

Limitations

Our proxies are not designed to query a single website on a high volume or request rate. We limit request rate to a single website and block certain ports to prevent overloading of websites, sending spam or other illegal activity. If you want to run a bot to automatically query Google or similar websites all available IPs will be banned with CAPTCHA pretty soon and it will not work.

Use our datacenter proxies to change the origin location or your requests, avoid blocks by geolocation, speed up regular data updates, etc.

Please also check out Acceptable Use Policy. We ask that you not engage in prohibited activity on our network.

Demo Screencast

Watch the screencast to get the idea of using BotProxy username API to control your outgoing IP address. Please note that control headers are not supported anymore.

Advanced Topics

Country and Location selection

By default we create a proxy user that has access to all locations available to your account except for OpenProxies. You can limit the locations using proxy user details form. Enter a comma separated list of location codes you want to use. For example if you want to make requests only from US locations enter us-ny,us-io,us-ca. To allow all locations except some add a minus sign in front of locations list. For example to exclude OpenProxies locations enter -xo.

You can have multiple proxy users on your account each with different available locations and other params.

There is also the possibility to adjust the country and location on each request. Read the Username API section for more details.

Username API

We provide username API to control outgoing locations using only proxy authentication credentials. To achieve this you should add special suffixes to your proxy user login as follows. Let assume your proxy user login is pxu1000-0

• Specify COUNTRY by adding +CN suffix to login. For example to limit outgoing locations to US only use pxu1000-0+US as your proxy user login
• Specify LOCATION by adding +loc suffix: use pxu1000-0+us-fl as your proxy username
• Add SESSION identifier to country or location by adding one more suffix as follows: pxu1000-0+US+SESSIONID or pxu1000-0+us-fl+SESSIONID. If you do not want to limit country or location but want to use session leave country/location blank and specify session as follows: pxu1000-0++SESSIONID

The complete usage example to route outgoing traffic through US locations only and within session 123456 is:

$ curl --proxy x.botproxy.net:8080 --proxy-user "pxu1000-0+us+123456:password" "https://httpbin.org/ip"
      

Locations

Currently we have proxy peers in datacenters in the following locations:

You can find a list of locations available according with your current subscription in your account page as well as all current proxy IP addresses for each location.

Sessions: IP persistence

In many cases you may want to run several parallel sessions from the same client, use different outgoing IP per session, and be able to change them when needed. This can be done by utilizing sessions. Sessions provide you fine grained control over outgoing IP addresses.

Each request going through a proxy uses a session. Even when you do not provide session ID a so called default session is used. Session is a combination of proxy user login, country or location and a session ID. Only proxy user login is required, if other parameters are not provided they are added as blank strings to create internal session identifier in superproxy. Sessions are not shared between superproxies. Each session has an IP address. IP address is assigned to a session on session creation and does not change during session lifetime. Until you keep using the session and it is valid, the IP address stays the same.

Sessions has a maximum session age that can be set for each proxy user separately in user details form. When a session gets older it is invalidated and even if you provide the same location and session ID a new session with a new IP is created. You can set maximum session age to 0 to remove session age limit restriction. All sessions are invalidated after 5 minutes of inactivity.

Summary:

1. IP is persistent within one Session until session Age Limit is reached.
2. Session is login+country or location+session_id
3. Each time Session is created a new IP is assigned as follows:
   • We take all IP addresses available to your proxy user (considering Limit Locations user settings)
   • Filter IPs to match requested country or location
   • Randomly select IP address from the remaining IPs
4. When session age limit is reached IP is renewed
5. If session is not used for 5 minutes it is deleted

Generate the random number on thread startup, and change it when you want to change the proxy peer assigned for the thread connection. Session ID can be any random string/counter: requests with the same session will use the same proxy peer (as long as possible).

To force an IP change, just modify the session ID.

HTTP Headers

We return outgoing proxy IP address in HTTP response headers. For HTTPS requests response headers are added immediately after HTTP/1.1 200 Connection established response from proxy:

CONNECT httpbin.org:443 HTTP/1.1
Proxy-Authorization: Basic bG90...

HTTP/1.0 200 Connection established
X-RPOXY-IP: xx.xx.xx.xx
[encrypted data]
      

Errors

Proxy can respond only with the following error codes. All other codes are returned from the website you are connecting to:

Open Proxies

We maintain a list of known open proxy servers. If you choose "OpenProxies" location your request will be routed through one of these proxies. Your requests are forwarded through less reliable & higher latency servers. Open proxies IPs are located in many different countries around the world. These open proxy IPs typically do not stay online very long, and are not operated on reliable infrastructure. The tradeoff for this lack of reliability is a huge increase in quantity & variability of IP addresses.

The list is regularly updated and contains around 1000 working IP addresses around the world. If any request through an open proxy fails due to a proxy error, that error is recorded, and the request is re-tried up to 3 more times, using a different proxy for each retry. Failed proxy is immediately removed from the list.

By default Open Proxies location is disabled on a proxy user. You need to manually enable it in your account dashboard.

OpenProxies location code is xo. You can additionally specify the desired country by adding 2 letter ISO country code after a minus sign: xo-us (example location code for US).

Secure Proxy (HTTPS/SSL)

A secure web proxy is a web proxy that the browser (or your other software) communicates with via encrypted connection, as opposed to clear text. In insecure public networks, such as airports or cafes, browsing over HTTP may leave the user vulnerable to cookie stealing, session hijacking or worse. A secure web proxy can add a significant layer of defense in these cases. Diagrams below explain different scenarios in detail. Please note: not all clients (browsers) support secure proxies. Refer your client documentation and set HTTPS proxy as needed.

We provide a secure proxy at the following address: x.botproxy.net:8443 You can connect using the same proxy users.

To use secure proxy in your applications you need to first establish a TLS connection to our secure proxy and then initiate regular HTTP proxy connection over established encrypted connection. Some other programs and libraries have built in support for secure proxies such as curl or Go http client.

Find below an explanation of how requests go when using different kind of proxies:

• HTTP/HTTP
• HTTPS/HTTP
• HTTP/HTTPS
• HTTPS/HTTPS

HTTP request using HTTP proxy

In this case request is sent from your client in plain text thus allowing your local network administrator or an attacker to intercept it and access all the data being sent. This is the least secure option.

HTTPS request using HTTP proxy

Target host name alongside your BotProxy proxy user credentials are transferred in plain text. Your client requests to create a tunnel to the specific host and port with CONNECT command and then makes secure communication via this tunnel. Parties on local network cannot intercept or hijack any data being transferred between you and remote host.

HTTP request using Secure (HTTPS) proxy

In this case your client establishes secure connection with our proxy server and no one can know what data is being transferred between you and target host. Moreover nobody in local network can know what is the target host. Even though the target host can be accessed using plain text HTTP protocol.

HTTPS request using Secure (HTTPS) proxy

Requests and data are encrypted all the way through. Most secure option.

Additional Documentation and Examples

• Web Scraping Guide
• Using Selenium Chromedriver with Rotating proxies
• Tunneling TCP traffic through HTTP proxy
• Using Puppeteer with Rotating Proxies
• Using Scrapy with Proxies (IP Rotating Proxy)
• Proxy Related How To Q&A