How to Set up a Proxy in Headless Browsers – A Guide for Developers

Collecting public data, testing interfaces, and checking the health of modern services all require a stable network infrastructure. Servers can detect scripts that send hundreds of concurrent requests. Nearly half of internet traffic is now automated. As a result, filtering systems on web resources are becoming more sophisticated. In this article, we will go over how to set up a proxy to maintain a stable connection and avoid crashes in your scripts.

Prerequisites

If you are a developer and want to know about the best tools, a comprehensive article on the latest web scraping frameworks (Selenium, Playwright, Scrapy) will be helpful.

Before You Set up a Proxy: SOCKS5 vs HTTP for Automation

Before integrating intermediary servers into the code, you need to choose a protocol first. Most commonly, developers choose between HTTP and SOCKS5. They operate at different levels of the OSI model and have distinct characteristics.

• HTTP is an application layer (L7) protocol. Can understand HTTP traffic, analyze and modify headers, cache information, etc. This helps to make web requests easier to automate.
• SOCKS5 works at the transport layer (L4). They are universal without inspecting packet contents and only pass TCP/UDP traffic. This provides increased performance and flexibility with complex network configurations.

Let's take a look at them together:

Before working out how to set up a proxy in your code, choose the type of protocol. If it is a simple text scrape, then HTTP is probably the best; for heavy media transfers or non-standard ports, SOCKS5 is the more promising choice.

Appropriate Choice of IP Addresses for Developers

In addition to the protocol, the actual type of address is also crucial. The reality is that it comes down to picking between residential vs datacenter proxies for scraping and automation.

• Datacenter IPs are created on cloud servers. They're low-cost and fast to set up, yet simple to detect, as they come from well-known hosting providers.
• Residential ISPs provide residential IPs. Have maximum trust but are more expensive.
• Mobile IPs are allocated from a pool of addresses provided by the mobile network operators. This is the safest choice, as thousands of real users can share a single mobile IP.

If it's a complex enterprise system, you should buy mobile proxy. With mobile proxy solutions for scraping projects, you can get your connection error rate to the lowest possible level.

As businesses grow, developers need to adjust their infrastructure. If you're faced with the decision of choosing a solid provider or migrating your current automation systems, it's worthwhile to read this proxy migration guide to move between the major providers without disrupting production. If you know how to set up nodes with a seamless migration in mind, you won't lose any money or get bogged down in technicalities.

Step‑by‑Step Examples: How to Set up a Proxy in Your Code

Time to get on to the practical level. We'll look at how to set up a proxy in the three most popular automation frameworks: Selenium, Playwright and Puppeteer.

1. Selenium WebDriver proxy configuration

Assuming that Selenium is your primary tool, you just pass the connection settings through to the driver configuration. If you use a Python headless browser, the configuration is as follows:

from selenium import webdriver
from selenium.webdriver.chrome.options import Options


# Initialize Chrome options
options = Options()

# Enable headless mode
options.add_argument("--headless=new")

# Configure the proxy server
proxy_server = "http://username:password@your_proxy_ip:port"
options.add_argument(f"--proxy-server={proxy_server}")

# Launch Chrome with the configured options
driver = webdriver.Chrome(options=options)

try:
    # Visit a test service to verify the browser's public IP
    driver.get("https://httpbin.org/ip")

    print("Current browser IP:")
    print(driver.page_source)
finally:
    driver.quit()

If you set up a proxy in Selenium WebDriver, all traffic will go through a special intermediary server.

2. Playwright web scraping and working with contexts

Playwright is a great new library, compared to the previous ones. With Playwright web scraping, you can have isolated contexts.

Let's see what it looks like to create a Playwright browser context proxy in a Node.js environment:

const { chromium } = require("playwright");

(async () => {
  // Launch the browser in headless mode
  const browser = await chromium.launch({
    headless: true,

    // Configure the proxy in the launch options
    proxy: {
      server: "http://your_proxy_ip:port",
      username: "your_proxy_username",
      password: "your_proxy_password",
    },
  });

  const context = await browser.newContext();
  const page = await context.newPage();

  try {
    await page.goto("https://httpbin.org/ip");

    const response = await page.textContent("body");

    console.log("Playwright connection details:", response);
  } catch (error) {
    console.error("Connection error:", error);
  } finally {
    await browser.close();
  }
})();

This is useful since the authentication information goes directly into the configuration object.

3. Proxy authentication in Puppeteer and IP rotation

Puppeteer is a bit different. Passing the username and password in the --proxy-server argument is not always supported by Chromium, so authentication in Puppeteer is done via a special asynchronous function on the page.

Here's an example of code that also employs dynamic rotating proxies in Node.js:

const puppeteer = require("puppeteer");

(async () => {
  const proxyIpPort = "http://your_proxy_ip:port";

  // Initialize the browser instance
  const browser = await puppeteer.launch({
    headless: true,
    args: [`--proxy-server=${proxyIpPort}`],
  });

  const page = await browser.newPage();

  // Configure proxy authentication
  await page.authenticate({
    username: "your_proxy_username",
    password: "your_proxy_password",
  });

  try {
    await page.goto("https://httpbin.org/ip");

    const content = await page.evaluate(() => document.body.innerText);

    console.log(
      "Puppeteer authenticated connection details:",
      content
    );
  } catch (error) {
    console.error("Puppeteer execution error:", error);
  } finally {
    await browser.close();
  }
})();

With a scheme such as this, you can be assured that your automated sessions are secured with authentication information.

Advanced Techniques for Optimizing Your Infrastructure

Just knowing how to set up a proxy is not sufficient to create a successful production system. You have to consider the behavior of web servers and their defenses.

Session persistence across proxy rotation

If the script changes its IP address frequently, the security system may consider it abnormal activity and terminate the active session. To ensure stability, enable session persistence. Before refreshing the IP, export the current cookies and local storage, and import them into the new browser context.

Anti-detect measures in headless Chrome

Some of the automated browsers have default characteristics (such as avigator.webdriver: true). Use anti‑detect techniques like plugging in masking tools (e.g., puppeteer-extra-plugin-stealth or other Playwright anti‑detect tools). They modify system variables, and your script appears as a normal user.

CAPTCHA handling with rotating IPs

If a system thinks that it is being accessed automatically, it will display a graphical test. The key to handling CAPTCHAs is to not have to deal with them at all. These checks are reduced to almost zero when high-quality residential and mobile addresses with high IP trust scores are used.

Wrapping Up

A basic ability for a modern engineer is to know how to set up a proxy in automated scripts. The right protocols (HTTP/SOCKS5), address types (residential or mobile), and reliable code will ensure that your infrastructure continues to operate seamlessly in all situations. Adjust your configurations and adhere to best development practices to achieve the same results.