How to Add and Modify HTTP Headers in Selenium: Techniques and Examples

Home Guide How to Add and Modify HTTP Headers in Selenium: Techniques and Examples

How to Add and Modify HTTP Headers in Selenium: Techniques and Examples

HTTP Selenium headers are essential for web communication as they establish request and response processes. These provide information for authentication, session management, security protection, and API functions.

Modifying HTTP headers becomes necessary in Selenium automation to build different user simulation environments for managing authentication mechanisms, controlling cookies, and performing efficient API testing.

This article provides multiple effective methods to add and modify HTTP headers in Selenium.

What are HTTP Headers?

Both client and server systems use HTTP headers as key-value pair components when connecting through HTTP request-response operations. The directives built into HTTP specify all relevant rules regarding data processing and delivery methods.

The four primary categories of HTTP headers exist as follows:

General Headers: They have dual functionality across requests and responses but do not contain data about the content.
Request Headers: The Request Headers transmit client information, including authentication data, user-agent parameters, and preferred data format.
Response Headers: Convey metadata about the response, such as cache settings and security policies.
Entity Headers: The entity headers specify the characteristics of the response body, encoding information, content type specification, and content length measurement.

Why are HTTP Headers Important?

Modifying HTTP headers in Selenium enables testers to simulate various client behaviors and evaluate application responses under different conditions.

Authentication Testing: Adding authorization headers automates authentication, reducing manual effort and speeding up API and web service testing.
Simulating Different User Agents: Modifying the User-Agent header ensures web applications display correctly across different devices and browsers.
Web Scraping and Anti-Bot Evasion: Adjusting headers like User-Agent, Referer, and Accept-Language helps mimic real user behavior and bypass bot detection.
Session and Cookie Management: Altering cookie headers allows testers to simulate different session durations and evaluate multi-user session handling.
API Testing: Customizing headers such as Authorization tokens ensures secure API access and verifies responses under different authentication scenarios.
Localization Testing: Modifying the Accept-Language header ensures accurate language selection and content localization for multilingual applications.

Also Read: How to perform Web Scraping using Selenium C#

Common HTTP Headers that are Used in Web Automation

Web automation requires HTTP headers to create efficient communication links between clients and servers.

Web application testing operations succeed by modifying these Selenium headers, which permit testers to generate testing situations that verify application responses under multiple circumstances.

Also Read: Getting Started with Website Test Automation

Several HTTP headers commonly appear in web automation systems, including the following list:

User-Agent

The User-Agent header reveals information about the client software, operating system, browser, and device.

Testers can modify this header to make their testing device emulation more specific by replicating different browser and device types, which helps verify application display accuracy across diverse platforms.

Example:

User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/122.0.0.0 Safari/537.36

Referer

The URL of the earlier webpage sends its location through the Referer header to the server. Everyone uses the header to monitor and analyze movements on websites. When performing tests, the Referer header becomes essential to simulate different website navigation paths and evaluate application behavior regarding referral systems.

Example:

Referer: https://www.google.com/

Authorization

A server authenticates user agents with secured resources through the Authorization header containing necessary user credentials. The system provides authentication mechanisms, including Basic, Bearer, and OAuth protocols.

When they add this header, automated testers can skip manual login processes and test authenticated endpoints quickly.

Example:

Authorization: Bearer abcdef1234567890

Accept-Language

The Accept-Language HTTP header enables the specification of user-selected languages when a client needs content to render. Testing multilingual applications requires adjusting this header since it determines how content gets localized according to user-selected languages.

Example:

Accept-Language: en-US,en;q=0.9

Accept-Encoding

A client declares the supported content encodings through the Accept-Encoding header. Testers can check server response compression functionality by adjusting this header as it verifies proper compression, which leads to enhanced bandwidth efficiency and performance.

Example:

Accept-Encoding: gzip, deflate

Cookie

Through the Cookie header, the client sends previously saved cookies to the server, which enables the maintenance of user sessions and stored preferences. A tester can use automation to manipulate this header to run various user sessions simultaneously and perform stateful interaction tests on the application.

Example:

Cookie: sessionId=abc123; theme=light

Common Use Cases for Modifying HTTP Headers in Selenium

Selenium requires modified HTTP headers to enable the accurate simulation of multiple client requests in different situations for web application testing.

The following list includes standard usage scenarios for HTTP header modification in Selenium testing:

Simulating Requests from Different Devices and Browsers: Changing the User-Agent header helps verify cross-platform compatibility and ensure correct rendering across multiple environments.
Handling Authentication: Using the Authorization header automates login processes, simplifying the testing of authenticated pages and APIs.
Modifying Cookies: Adjusting cookie headers enables testers to evaluate session management, persistent logins, and security features.
API Testing: Customizing request headers allows authentication token management and validation of API responses under different conditions.

How to Add HTTP Headers in Selenium WebDriver

Multiple strategies exist for integrating header modification functionality. Here, you will find the most effective procedures for including HTTP headers using Selenium WebDriver.

Must Read: How to Download and Set Up Selenium WebDriver for Your Automation Projects

1. Using Chrome DevTools Protocol (CDP)

The Chrome DevTools Protocol (CDP) offers developers tools for direct network intervention, including the capability to edit HTTP headers.

Implementation:

import org.openqa.selenium.devtools.DevTools;

import org.openqa.selenium.devtools.v85.network.Network;

import org.openqa.selenium.devtools.v85.network.model.Headers;

import org.openqa.selenium.chrome.ChromeDriver;

import java.util.HashMap;

import java.util.Map;




public class ModifyHeadersUsingCDP {

    public static void main(String[] args) {

        ChromeDriver driver = new ChromeDriver();

        

        // Create a DevTools session

        DevTools devTools = driver.getDevTools();

        devTools.createSession();




        // Define custom headers

        Map<String, String> headers = new HashMap<>();

        headers.put("User-Agent", "Custom User Agent");

        headers.put("Authorization", "Bearer your_token_here");




        // Apply headers using CDP

        devTools.send(Network.setExtraHTTPHeaders(new Headers(headers)));




        // Navigate to the test website

        driver.get("https://bstackdemo.com");




        // Close the browser

        driver.quit();

    }

}

Pros:

No external dependencies required
Works seamlessly with Chrome

Cons:

Limited to Chrome and may break with version updates

Also Read: How to run Selenium tests on Chrome using ChromeDriver?

2. Using BrowserMob Proxy

The HTTP proxy functionality of BrowserMob Proxy enables change requests en route to the server before delivery.

Implementation:

import net.lightbody.bmp.BrowserMobProxy;

import net.lightbody.bmp.BrowserMobProxyServer;

import net.lightbody.bmp.client.ClientUtil;

import org.openqa.selenium.Proxy;

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.chrome.ChromeOptions;




public class ModifyHeadersUsingBrowserMobProxy {

    public static void main(String[] args) {

        // Start the BrowserMob Proxy

        BrowserMobProxy proxy = new BrowserMobProxyServer();

        proxy.start(0);




        // Add custom headers

        proxy.addRequestFilter((request, contents, messageInfo) -> {

            request.headers().add("User-Agent", "Custom User Agent");

            request.headers().add("Authorization", "Bearer your_token_here");

            return null;

        });




        // Configure Selenium to use the proxy

        Proxy seleniumProxy = ClientUtil.createSeleniumProxy(proxy);

        ChromeOptions options = new ChromeOptions();

        options.setProxy(seleniumProxy);




        // Initialize WebDriver with proxy settings

        WebDriver driver = new ChromeDriver(options);

        driver.get("https://bstackdemo.com");




        // Close WebDriver and stop proxy

        driver.quit();

        proxy.stop();

    }

}

Pros:

Works with multiple browsers
Allows traffic interception and modification

Cons:

Users must set up this solution while expense and performance degradation become possible side effects

3. Using REST Assured for API Testing

The REST Assured library provides a tool for executing HTTP requests that require customized headers during API testing.

Implementation:

import io.restassured.RestAssured;

import io.restassured.response.Response;




public class ModifyHeadersUsingRestAssured {

    public static void main(String[] args) {

        Response response = RestAssured.given()

            .header("User-Agent", "Custom User Agent")

            .header("Authorization", "Bearer your_token_here")

            .get("https://bstackdemo.com");

        

        // Print response for verification

        System.out.println(response.getStatusCode());

    }

}

Pros:

Ideal for API testing
Lightweight and efficient

Cons:

This tool does not work for controlling User Interface elements or communicating through web interface components

Must Read: Top 15 API Testing tools

4. Using ModHeader Chrome Extension

Users can edit request headers through the ModHeader Chrome extension without needing either proxies or CDP technology.

Implementation:

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.chrome.ChromeDriver;

import org.openqa.selenium.chrome.ChromeOptions;

import java.io.File;




public class ModifyHeadersUsingModHeader {

    public static void main(String[] args) {

        // Load the ModHeader extension

        ChromeOptions options = new ChromeOptions();

        options.addExtensions(new File("path/to/ModHeader.crx"));




        // Initialize WebDriver with the extension

        WebDriver driver = new ChromeDriver(options);




        // Apply headers via ModHeader extension settings

        driver.get("https://webdriver.modheader.com/add?User-Agent=Custom%20User%20Agent&Authorization=Bearer%20your_token_here");

        

        // Navigate to test website

        driver.get("https://bstackdemo.com");




        // Close browser

        driver.quit();

    }

}

Pros:

No coding is required to modify headers
It can be used with headless Chrome

Learn More: Headless Browser Testing With Selenium Python

Cons:

Requires manual installation of the extension

Method	Best For	Dependencies	Supported Browsers
Chrome DevTools Protocol (CDP)	UI testing in Chrome	None	Chrome only
BrowserMob Proxy	Cross-browser testing & traffic interception	BrowserMob Proxy	Chrome, Firefox, Edge
REST Assured	API testing	REST Assured Library	Not browser-dependent
ModHeader Extension	Quick header modifications without code	ModHeader Extension	Chrome

Best Practices for Modifying HTTP Headers in Selenium

Modifying HTTP headers through Selenium WebDriver serves many useful purposes, including user simulation authentication testing and API interaction management. These best practices lead to strong maintainable test automation when used to modify HTTP headers with Selenium:

1. Maintain Test Isolation

Each test should run independently without relying on data from previous tests. To avoid contamination, reset or clear modified headers before every test execution. Use setup and teardown methods to initialize and clean up the testing environment efficiently.

Example:

@Before

public void setup() {

    // Set up browser and headers

}




@After

public void teardown() {

    // Close browser to ensure clean state

    driver.quit();

}

2. Handle Sensitive Data Securely

To prevent security risks, avoid storing API tokens and credentials directly in test scripts. Store sensitive data in configuration files or environment variables, ensuring restricted access. Use secure credential managers or encryption methods to protect critical information.

Example (Using Environment Variables for Authentication Tokens):

String authToken = System.getenv("AUTH_TOKEN"); 

headers.put("Authorization", "Bearer " + authToken);

3. Verify Headers in Tests

Ensure headers are correctly modified by using assertion tests during execution. Validate network responses to confirm server recognition of applied headers. Monitor and verify request headers using BrowserMob Proxy.

Also Read: How to set Proxy in Selenium?

Example (Using REST Assured for Verification):

Response response = RestAssured.given()

    .header("User-Agent", "Custom User Agent")

    .header("Authorization", "Bearer " + System.getenv("AUTH_TOKEN"))

    .get("https://example.com");




assert response.getHeaders().hasHeaderWithName("User-Agent");

assert response.getStatusCode() == 200;

4. Use the Right Method for the Use Case

For Chrome-based UI testing, Chrome DevTools Protocol (CDP) is the best choice. BrowserMob Proxy is suitable for server network interception and cross-browser testing. REST Assured is ideal for API testing, while ModHeader Extension allows quick manual modifications of request headers.

5. Keep Test Execution Efficient

Modify headers only when necessary for testing requirements. Manage BrowserMob Proxy start and stop commands to prevent performance delays. Reuse API sessions to reduce unnecessary network requests.

Why Run Selenium Tests on BrowserStack Automate?

BrowserStack Automate provides robust, scalable, and real user conditions for running Selenium tests. This ensures faster, more accurate, and seamless automation testing.

Real Device Cloud: Run Selenium tests on 3,500+ real devices and browsers to simulate real-world user conditions and uncover critical bugs.
Cross Browser Testing: Ensure compatibility across multiple browser-OS combinations, including legacy and latest versions, without setup hassles.
Parallel Test Execution: Speed up test execution with parallel testing, reducing test cycle times and accelerating release velocity.
Seamless CI/CD Integration: Integrate effortlessly with Jenkins, GitHub Actions, CircleCI, and more for continuous testing in your development pipeline.
Advanced Debugging Tools: Get instant access to logs, screenshots, and video recordings for faster issue identification and resolution.

Talk to an Expert

Useful Resources for Selenium

Methods, Classes, and Commands

Configuration

XPath

Locators and Selectors

Waits in Selenium

Frameworks in Selenium

Miscellaneous

Best Practices, Tips and Tricks

Design Patterns in Selenium: Page Object Model and Page Factory

Action Class

TestNG and Selenium

JUnit and Selenium

Use Cases

Types of Testing with Selenium

Conclusion

Through HTTP Selenium headers modifications, users achieve essential features such as authentication verification and API examination and conduct accurate emulation of user sessions.

The native header modification capability of Selenium WebDriver is absent from its core features. Still, testers can use alternative solutions like REST Assured, BrowserMob Proxy, and ModHeader extensions to accomplish this task.

The cloud-based testing environment of BrowserStack Automate provides testers with an efficient solution to execute Selenium tests over various browsers and devices.

Automation Tests on Real Devices & Browsers

Seamlessly Run Automation Tests on 3500+ real Devices & Browsers

How to Add and Modify HTTP Headers in Selenium: Techniques and Examples