Setting chromedriver proxy with Selenium using Python

If you need to use proxy with python and Selenium library with chromedriver you usually use the following code:

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=%s' % hostname + ":" + port)
driver = webdriver.Chrome(chrome_options=chrome_options)

This setup works well for basic proxies without authentication. However, if your proxy requires a username and password (login with a username and password), additional steps are necessary. Below, we’ll explore how to set up an authenticated proxy.

Using BotProxy for Authenticated Proxies

If you’re leveraging BotProxy as your proxy for web scraping, you can whitelist your server’s IP address to bypass HTTP authentication. However, for scenarios requiring username and password authentication, follow these steps:

By the way if you are going to use BotProxy rotating proxy you can whitelist your server IP address and botproxy will not require HTTP auth to work.

Setting Up HTTP Proxy Authentication with Chromedriver in Selenium

To configure Selenium to use an authenticated HTTP proxy, we’ll dynamically create a Chrome extension to manage the proxy settings. The example below is given for our BotProxy Web Scraping Proxy server, but you can substitute PROXY_HOST and other constants with your values. This code configures selenium with chromedriver to use HTTP proxy that requires authentication with user/password pair.

import os
import zipfile

from selenium import webdriver

PROXY_HOST = 'x.botproxy.net'  # rotating  web scraping proxy
PROXY_PORT = 8080
PROXY_USER = 'proxy-user'
PROXY_PASS = 'proxy-password'


manifest_json = """
{
    "version": "1.0.0",
    "manifest_version": 2,
    "name": "Chrome Proxy",
    "permissions": [
        "proxy",
        "tabs",
        "unlimitedStorage",
        "storage",
        "<all_urls>",
        "webRequest",
        "webRequestBlocking"
    ],
    "background": {
        "scripts": ["background.js"]
    },
    "minimum_chrome_version":"22.0.0"
}
"""

background_js = """
var config = {
        mode: "fixed_servers",
        rules: {
          singleProxy: {
            scheme: "http",
            host: "%s",
            port: parseInt(%s)
          },
          bypassList: ["localhost"]
        }
      };

chrome.proxy.settings.set({value: config, scope: "regular"}, function() {});

function callbackFn(details) {
    return {
        authCredentials: {
            username: "%s",
            password: "%s"
        }
    };
}

chrome.webRequest.onAuthRequired.addListener(
            callbackFn,
            {urls: ["<all_urls>"]},
            ['blocking']
);
""" % (PROXY_HOST, PROXY_PORT, PROXY_USER, PROXY_PASS)


def get_chromedriver(use_proxy=False, user_agent=None):
    path = os.path.dirname(os.path.abspath(__file__))
    chrome_options = webdriver.ChromeOptions()
    if use_proxy:
        pluginfile = 'proxy_auth_plugin.zip'

        with zipfile.ZipFile(pluginfile, 'w') as zp:
            zp.writestr("manifest.json", manifest_json)
            zp.writestr("background.js", background_js)
        chrome_options.add_extension(pluginfile)
    if user_agent:
        chrome_options.add_argument('--user-agent=%s' % user_agent)
    driver = webdriver.Chrome(
        os.path.join(path, 'chromedriver'),
        chrome_options=chrome_options)
    return driver

def main():
    driver = get_chromedriver(use_proxy=True)
    #driver.get('https://www.google.com/search?q=my+ip+address')
    driver.get('https://httpbin.org/ip')

if __name__ == '__main__':
    main()

Function get_chromedriver returns configured selenium webdriver that you can use in your application. This code is tested and works just fine with BotProxy. When using BotProxy inside Chrome or Chromedriver make sure to disable Bot Anti-Detect mode since with real browser it has correct TLS fingerprints itself.

Key Considerations

  1. Proxy Authentication: The dynamically created extension handles proxy authentication seamlessly.

  2. BotProxy: Disable the Bot Anti-Detect mode when using a real browser, as the browser already has proper TLS fingerprints.

  3. Testing: Test your setup by visiting websites like httpbin.org/ip to verify the proxy is correctly configured.

With this approach, you’re ready to handle authenticated proxies in Selenium with Chromedriver, making your web scraping and automation tasks more flexible and secure.