How to set a Custom User Agent in Selenium?

A User Agent is a header string that identifies the system, browser, and device when making a request to a server. It helps websites render optimized content by detecting mobile or desktop environments and adjusting images, menus, and styles for better responsiveness.

By adapting content based on browser-device combinations, User Agents ensure smooth web application performance across different platforms.

This guide explains user agents, their significance, how to set and test them, best practices, and more.

What is a User Agent?

The user agent is a string of text containing information about the browser, device, version, etc. The user agent string is sent as a header while sending an HTTP request to the server. It allows smoother functionality by adjusting the content to the specific user agents.

A User agent typically contains the information below:

• Browser: Type of the Browser such as Chrome, Firefox, Safari, etc.
• Browser Version: The version of the browser used.
• Operating System: The operating system name such as Windows, macOS, Android, iOS, etc.
• Device Type: Type of the device such as mobile, desktop, tablet, etc.

Example of User agent

Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36

The above user agent string provides the below information

• Browser: Chrome
• Version:91.0
• Rendering Engine: WebKit

The Impact of a User Agent on Website Rendering and Behavior

User agents are very useful for responsive web pages and provide a smooth user experience. Typically, content delivery will be optimized based on the device and browser types. It has the following advantages:

• Smooth Rendering: The user agents help to display optimized content based on the device. For example, if the user agent has a mobile device indication, websites render a mobile-friendly version instead of a desktop version.
• Browser-specific features: It allows the browser-specific features based on the browser type detected. For example, Chrome might offer many different types of animations, while Firefox does not. By knowing the browser type, it is easy to deliver the content specific to the browser.
• Operating System-Specific Features: The native interfaces may change based on the operating system. The user-agent helps to render and tune the interfaces according to the operating system that you are using.

Why do you need to set a Custom User Agent in Selenium?

Selenium is a p popular test automation framework that provides flexibility to the tester. For example, the tester can customize the browsers, user agents, and responsiveness.

User agents help to test the responsiveness and many other features using test automation. Below pointers explain the need for custom user agents in Selenium.

1. Responsive Design Testing: The responsive design approach changes the webpage layout based on the device type. It adjusts the screen width and height of the device.

The custom user agent simulates the website layout on various devices, such as mobile, tablet, or desktop, and helps ensure it works without breaking.

2. Test functionality: Some features are available based on the device. However, it may be difficult to test all the features using physical devices.

The user agents help in simulating specific devices and versions, such as Android with Samsung Galaxy, iPhone 12, iPad, etc., using the desktop.

3. Bypassing: Custom user agents also help bypass specific restrictions based on the device or browser type.

For example, some content may be blocked on mobile websites to ensure a smooth experience. By sending the custom user agents, you can unblock such restrictions and test them.

4. Testing compatibility: While it may be difficult to install the different versions of the browser, by setting the custom user agents you can ensure that websites are compatible with older versions of the browsers.

5. Language/Location Specific features: The website may be available in various languages by default it may show the specific language based on the region. These features can be simulated using the user agents.

How to set a Custom User Agent in Selenium WebDriver?

Selenium allows to set the custom user agents. The browser driver can be configured to test custom user agents. Using Selenium custom user agents can be set for any browser such as Chrome, Edge, Firefox, etc.

Set custom user agents for Chrome using Selenium

  public void testChrome(){

        ChromeOptions options = new ChromeOptions();

        options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36");

        WebDriver driver = new ChromeDriver(options);

        driver.get("https://bstackdemo.com/");

        driver.close();

    }

In the above example, a custom user agent is set for Chrome, and a ChromeOptions object is created and named as an option.

Then, an argument is added as a user agent to set the user agent using the options.addArguments() method.

Finally, a new Driver object is created with the specified options using new ChromeDriver(options);.

Set custom user agents for Firefox using Selenium

    @Test

    public void testFirefox() {

        FirefoxProfile profile = new FirefoxProfile();

        profile.setPreference("general.useragent.override", "Mozilla/5.0 (Android; Mobile) Gecko/68.0 Firefox/68.0 QwantMobile/3.5");

        FirefoxOptions options = new FirefoxOptions();

        options.setProfile(profile);

        WebDriver driver = new FirefoxDriver(options);

        driver.get("https://bstackdemo.com/");        driver.close();

    }

In the above example, a custom user agent is set for Firefox.

A FirefoxProfile object is created to set the preference.

Using the FirefoxProfile object, the user agent is overridden with the code profile.setPreference(“general.useragent.override”, “”);.

Then, you create the FirefoxOptions objects to set the above profile (options.setProfile()).

Finally, you can create the FirefoxDriver object with these options.

Set custom user agents for Edge using Selenium

Like Firefox and Chrome, You can also set the custom user agents for Edge in Selenium.

    @Test

    public void testEdge(){

        EdgeOptions options = new EdgeOptions();

        options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36");

        WebDriver driver = new EdgeDriver(options);

        driver.get("https://bstackdemo.com/");

        driver.close();

    }

In the above code, the EdgeOptions object is selected to add the arguments.

The options.addArguments() method sets the user agent, and the options object is passed as a parameter to create the new Edge Driver.

How to Test the Custom User Agent?

There are different ways to verify the custom user agents. There are also several websites that displays the Browser information based on the agents that you have set.

Consider that you are using the Selenium code below.

import org.openqa.selenium.WebDriver;

import org.openqa.selenium.firefox.FirefoxDriver;

import org.openqa.selenium.firefox.FirefoxOptions;

import org.openqa.selenium.firefox.FirefoxProfile;

import org.testng.annotations.Test;

public class Demo {

    @Test

    public void testFirefox() {

        FirefoxOptions options = new FirefoxOptions();

        FirefoxProfile profile = new FirefoxProfile();

        profile.setPreference("general.useragent.override", "Mozilla/5.0 (Android; Mobile) Gecko/68.0 Firefox/68.0 QwantMobile/3.5");

        options.setProfile(profile);

        WebDriver driver = new FirefoxDriver(options);

        driver.get("https://www.whatismybrowser.com");

        driver.close();

    }

}

It can be verified that the custom user agents are set correctly on the website. In the above code, after the user agents are set, the website https://www.whatismybrowser.com is navigated to, where the browser name should be displayed as Android, as an Android and Qwant browser is being set.

Testing user agents manually

Once you set the user agents, you can manually check the network logs

RightClick > Inspect > Network > Check any request headers of server responses

Talk to an Expert

Use Cases for Custom User Agents in Selenium Testing

There are many cases where custom agents are used in Selenium. As mentioned before it helps to validate the responsiveness of the page, validate the device compatibility, verify the restrictions such as captcha, test cross-browser behavior, etc. Few of them are explained below.

Testing Mobile Sites

The server sends the response based on the user agent’s response. If you send the user agent an indication that the request is from mobile, the browser renders the mobile-specific content.

Example:

   public void testChromeMobile() throws InterruptedException {

        ChromeOptions options = new ChromeOptions();

        options.addArguments("user-agent=Mozilla/5.0 (Linux; Android 10; K) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0.0.0 Safari/537.36");

        WebDriver driver = new ChromeDriver(options);

        driver.get("https://www.google.com/search?q=Browserstack");

        driver.close();

    }

In the above example, the Android 10 Mobile user agent is set. When Google.com search is navigated to, the mobile version of Google.com is rendered, as shown below.

Bypassing Restrictions

Many websites will be restricted from scraping data when the user agent is identified as a bot. Typically, the bot will be used to scrape large amounts of data from various websites. Such restrictions can be bypassed by setting a custom user agent.

Execute the test by sending the bot user agent to understand the restrictions

    @Test

    public void testChrome() throws InterruptedException {

        ChromeOptions options = new ChromeOptions();

        options.addArguments("user-agent=Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)");

        WebDriver driver = new ChromeDriver(options);

        driver.get("https://www.medium.com");

        driver.close();

    }

In the above example, the bot user agent belonging to Bingbot is sent. Once the medium.com website is navigated to, the website is blocked as bots are not allowed to crawl the web page.

You can remove the restrictions by setting a Firefox user agent.

    @Test

    public void testChrome() throws InterruptedException {

        ChromeOptions options = new ChromeOptions();

        options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0");

        WebDriver driver = new ChromeDriver(options);

        driver.get("https://www.medium.com");

        driver.close();

    }

The above code sets the Firefox user agent to bypass the restrictions. Upon executing it, you will see that medium.com is loaded.

Testing Cross-Browser Behaviour

User agents can also be used to test the cross-browser behaviour of web applications. You can mimic the user agent of any browser.

Example: Sending the Safari User Agent on Chrome

    @Test

    public void testChrome() {

        ChromeOptions options = new ChromeOptions();

        options.addArguments("user-agent=Mozilla/5.0 (Windows; U; Windows NT 6.1; ko-KR) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27");

        WebDriver driver = new ChromeDriver(options);

        driver.get("https://www.browserstack.com");

        driver.close();

    }

In the above code, the Safari user agent is sent

Example: Sending the Firefox User Agent on Chrome

  @Test

    public void testChrome() {

        ChromeOptions options = new ChromeOptions();

        options.addArguments("user-agent=Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:135.0) Gecko/20100101 Firefox/135.0");

        WebDriver driver = new ChromeDriver(options);

        driver.get("https://www.browserstack.com");

        driver.close();

    }

The above code uses the Firefox browser user agent which is of version 135.0.

Best Practices for using Custom User Agents in Selenium

Here are some of the best practices for using custom user agents:

• Do not violate the terms of service: You must ensure that if you are bypassing restrictions, it is allowed to bypass. Scraping data without permission can violate legal agreements and may result in strict actions.
• Ensure the user agent is realistic: The user agent should be set in a realistic way such as device, version, and browser should be properly given.
• Simulate the Real User Behaviour: Rather than sending too many requests with random user agents, use conditions and delay to mimic the real user behavior.
• Use real devices to test: Rather than simulating mobile devices, different operating systems use cloud providers like BrowserStack to test your website on real devices. The user-agent-based simulations may not always be accurate. Using real devices helps to get the correct results.

Why choose BrowserStack to test custom user agents in Selenium?

While custom user agents can help simulate different devices and browsers, they do not fully replicate real-world conditions. Factors like screen resolution, operating system behavior, touch interactions, and hardware variations also impact website performance.

With BrowserStack Automate, you can test across real devices and browsers rather than relying solely on user-agent simulation. This gives accurate, reliable, and comprehensive test results. Additionally, Selenium integration is easier, allowing you to run tests on BrowserStack without modifying your existing scripts.

Conclusion

Custom user agents help validate web page content quickly and allow us to know how applications behave under different conditions.

However, considering end-to-end testing and compatibility testing, the custom user agents may not provide accurate results. BrowserStack is a cloud-based test execution platform that provides many real devices, including desktop, web, and mobile.

Additionally, you are flexible to choose the different browsers such as Chrome, Safari, Firefox, Edge, etc. BrowserStack Automate can also help to automate such scenarios by integrating popular test automation frameworks including Selenium.

Additionally, BrowserStack Automate can also perform cross browser testing, responsive testing, cross-device testing, compatibility testing, and functional testing.

Try BrowserStack Now