Understanding Puppeteer waitUntil

Puppeteer is a general-purpose automation framework that can be used for test automation, end-to-end browser-based testing, web scraping, etc. Puppeteer is developed by Google, and it provides a high-level API to automate browser-based scenarios.

It provides an easy way to simulate real user scenarios such as clicks, navigation, waits, etc. Puppeteer can capture screenshots and generate PDFs, and it can be used to validate progressive web applications (PWAs).

Puppeteer can be used with Node.js and JavaScript programming languages to automate the scenarios. Puppeteer provides many advanced features for waiting mechanisms, and the waitUntil option is one of the most powerful ways in Puppeteer; it waits based on the event triggered by the browser rather than element- or time-based wait.

What is Puppeteer Wait Until?

waitUntil is an option available in Puppeteer that can be used with the navigation method page.goto(). It allows users to specify the waiting event for the navigation. These events may be networkidle, load, or DOMContentLoaded.

Often, Puppeteer may try to perform an action by executing the next available command before the page loads. These events help to ensure that the page is loaded and specific events are triggered before performing any actions on the page.

Why use Puppeteer and Chrome for HTML conversion?

Puppeteer can handle complex web pages accurately, even if many CSS, JavaScript, and HTML resources are embedded within. Puppeteer provides API control over the Chrome browser, which helps to render, manipulate, and debug the captured content easily.

Puppeteer and Chrome render the page the same way an end user would see it in their browser. Additionally, Puppeteer is more automation-friendly as it provides many customization options, such as defining the viewport, specifying timeouts, waiting for specific events, screenshot options, PDFs, etc.

Navigating Pages in Puppeteer

In Puppeteer, navigation can be performed using the page.goto() method. This method loads the specified URL in the browser. The page.goto() method can accept two parameters: the first one is the URL of the web page, which is mandatory, and the second one is the waitUntil parameter, which is optional.

As mentioned earlier, the waitUntil option can be used to wait for specific events to be triggered during the page navigation. You can set events like ‘load‘, ‘DOMContentLoaded‘, ‘networkidle0‘, or ‘networkidle2‘, depending on the specific requirements.

Each option has its own purpose; for example, setting waitUntil: ‘networkidle0‘ ensures that all network activity has been completed, and this option might be ideal for single-page applications.

Challenges in Content Loading

Content loading in modern web pages has many challenges, including the lazy loading mechanism, network delays, etc. Such challenges may impact test automation or sometimes even the user experience.

Below are a few challenges in content loading:

• Slow Network Speed: Slow network speed can cause page loading delays, which may eventually throw timeout exceptions. When a webpage is made up of a large number of images, videos, or animations, this can become problematic.
• Dynamic Content: On dynamic content web pages, the content will be loaded after the initial page load. Ajax requests are sent after the initial page load. Typically, automation code expects elements to be available at the initial page load. However, to handle dynamic content, testers might need to implement special mechanisms.
• Single Page Applications: Modern applications built with React, Angular, etc., use the SPA mechanism, where content is loaded dynamically as the user performs actions. It may be difficult to identify when the events are triggered and when content is updated.
• Heavy Resource Rendering: If the website contains large videos, images, and complex JavaScript, this can slow down page rendering. When writing automation scripts, this can have a significant impact on execution time, as you may need to wait for the page to complete loading, even though some resources, like videos, may not be required for validation.

Understanding Puppeteer waitUntil

The waitUntil option is used to navigate to a specific URL. It can only be used with the page.goto() method. The waitUntil option can be set to one of the following values: ‘load‘, ‘networkidle0‘, ‘networkidle2‘, or ‘domcontentloaded‘.

Syntax:

page.goto(url, waitUntilOptions)

• page.goto(): method used for navigation
• url: It is a specific web page url
• waitUntilOptions: wait until options are specific in the format of waitUntil: ‘eventType’ where event type can be ‘load’, networkidle0, networkidle2, or domcontentloaded.

Example:

page.goto('https://bstackdemo.com/',{waitUntil: 'domcontentloaded'});

Why use waitUntil option in Puppeteer?

The waitUntil option in Puppeteer helps specify when the browser should consider the navigation as complete. This ensures that subsequent actions on the web page are performed accurately and reduces flakiness.

Modern web applications are often built with dynamic loading mechanisms, where content is loaded after the initial page load. If one tries to perform an action before the content has fully loaded, the automation test may fail because the elements may not be available at that moment.

The waitUntil options help eliminate such flakiness. Additionally, waitUntil can be instrumental in building reliable automation scripts.

Available Options for waitUntil in Puppeteer

As mentioned earlier, waitUntil has four different options, each with its own significance. Below are the waitUntil options:

1. ‘load’

This option waits for the load event. This signals that the entire page and all its resources such as images, stylesheets, javascript, etc. have been fully loaded.

2. ‘domcontentloaded’

This option waits for the DOMContentLoaded event, which occurs once the HTML document has been completely loaded and parsed. However, this does not guarantee that stylesheets, images, or any other resources are loaded.

3. ‘networkidle0’

This option waits for the network to be idle, meaning there are no active network connections for at least 500 milliseconds.

4. ‘networkidle2’

This option is similar to the above ”networkidle0′, but waits until there are no more than 2 active network connections for at least 500 milliseconds. This ignores if there are less than 2 background activities running.

Comparison of waitUntil Options in Puppeteer

Here is a comparison of the various options of the WaitUntil method in Puppeteer:

Examples of waitUntil Usage

In the above section, you got an idea about different options of waitUntil and some of the use cases. Now, understand in detail with a Puppeteer example.

Waiting for Full Page Load (load)

Consider an example where you are navigating to https://bstackdemo.com/ and you want to add the iPhone 12 Pro to the cart. However, this action can be performed only after the complete page is loaded in such cases, you can use the waitUntil with load option.

Example:

const puppeteer = require('puppeteer');

const { setTimeout } = require("node:timers/promises");

(async () => {

    const browser = await puppeteer.launch({ headless: false });

    const page = await browser.newPage();

    await page.goto('https://bstackdemo.com/', { waitUntil: 'load' })

    await page.click('div[data-sku*="iPhone12Pro-device-info.png"]>div[class="shelf-item__buy-btn"]');

    console.log("Item added to cart succesfully!")

    await setTimeout(10000);

    await browser.close();

})();

The above code is to launch the browser with UI mode by specifying headless: false.

Using page.goto(), navigate to https://bstackdemo.com/ and wait for all the resources to be loaded using the waitUntil: ‘load’ option. Once page loading is complete, click on the Add to Cart button.

Output

Waiting for DOM Ready (domcontentloaded)

Consider an example where you want to fetch the title of the web application. To fetch the title, you do not need to wait for all the resources to be loaded in such cases, you can use the domcontentloaded option.

Example

const puppeteer = require('puppeteer');

const { setTimeout } = require("node:timers/promises");

(async () => {

    const browser = await puppeteer.launch({ headless: false });

    const page = await browser.newPage();

    await page.goto('https://bstackdemo.com/', { waitUntil: 'domcontentloaded' })

    console.log("The page title is: "+await page.title())

    await setTimeout(10000);

    await browser.close();

})();

The above code launches the browser with UI mode by specifying headless: false.

Using page.goto() navigate to https://bstackdemo.com and wait for only dom content to be loaded but not all the resources using the waitUntil: ‘domcontentloaded’ option. Once DOM content is loaded fetch the page

Output

Waiting for Network Stability

Consider you have a website that dynamically loads the images after the initial load by sending the network requests in the backend. You may need to fetch the count of images. In such a scenario you can use networkidle0 or networkidle2 based on your preferences

const puppeteer = require('puppeteer');

(async () => {

    const browser = await puppeteer.launch({ headless: false });

    const page = await browser.newPage();

    //With networkidle0

    await page.goto('https://example.com/greenearth-images', { waitUntil: 'networkidle0' })

    //With networkidle2

    // await page.goto('https://example.com/greenearth-images',{waitUntil:'networkidle2'})

    const imageCount = await page.$$('div[class="modern-gen"]>img').length

    console.log(imageCount);

    await browser.close();

})();

The above example launches the browser in UI mode by specifying the headless: false.

Then navigate to the web page ‘https://example.com/greenearth-images‘ and wait for all the backend network calls to complete using networkidle0. If less than two network requests are acceptable, you can also use networkidle2 here. Once network calls are complete, fetch the image count using the page.$$().length function.

Handling Edge case scenarios

As mentioned in the previous sections, modern web applications use advanced technologies like lazy loading, where content is loaded after the initial page load, triggered by user actions such as scrolling to the bottom or clicking. Some of the edge case scenarios may include lazy-loaded images, lazy-loaded content based on scrolling, videos, or animated content.

These edge-case scenarios can be handled in Puppeteer with some workarounds. For instance, if you need to handle lazy-loaded content based on scrolling position, you may need to incorporate scrolling logic and network idle together. For animated content that loads after a click, you may need to implement a click action followed by the networkidle waitUntil option.

To handle such edge-case scenarios, Puppeteer provides a method called page.waitForNetworkIdle(). This method is similar to the earlier waitUntil: ‘networkidle0’ option but can be used independently and does not need to be used with page.goto()

Example Snippet

const puppeteer = require('puppeteer');

(async () => {

    const browser = await puppeteer.launch({ headless: false });

    const page = await browser.newPage();

    await page.goto('https://example.com/lazy-scrolling')

    await page.evaluate(() => window.scrollBy(0, window.innerHeight)),

    await page.waitForNetworkIdle();

    //Do some actions

    await browser.close();

})();

In the above code, after navigating to the web page, scroll the page using the window.scrollBy() function, and then wait for the network to be idle or complete the background activity.

Talk to an Expert

Choosing the Right waitUntil Option

Here is a guide to choosing the right waitUntil option:

• Use the ‘load‘ option when you want to load all the page content, including images, stylesheets, and JavaScript, such as clicking on links, typing in input boxes, etc.
• Use ‘domcontentloaded‘ when you want to load only DOM elements to be rendered and parsed, such as fetching the page title, URL, etc.
• Use ‘networkidle0‘ when you do not want any background activity to be present. For example, in a dynamic loading web page, if you want all the assets to be completely loaded.
• Use ‘networkidle2‘ when you are expecting less than two background activities to be running, such as web pages with real-time trends that may continuously poll for APIs to get metrics.

Why run Puppeteer tests on Real Devices?

As discussed above, the website experience may change based on various conditions like network speed, web page content, browser type, operating system, device, etc.

When shipping a product, ensure the web application provides a smoother experience to the user in all possible conditions. To simulate such scenarios, execute tests on real devices.

Executing Puppeteer tests on real devices has many benefits. Below are a few:

• Real devices help to simulate edge case scenarios such as network speed, and true user behavior, including rendering, touch gestures, etc.
• It can help identify any device or browser-specific issues that may cause problems in the production environment.
• It aids in assessing the performance of web pages under different environments and combinations.
• It allows capturing the user experience under different network conditions.

Why choose BrowserStack to run Puppeteer Tests?

Running a Puppeteer test on real devices is important to ensure the quality of the application under different scenarios.

However, creating such an environment on-premises is not easy. Organizations may need a dedicated team to manage and maintain the infrastructure for the long term. Additionally, meeting the varying demands is another challenge.

BrowserStack provides a scalable environment to execute the tests without worrying about creating and maintaining the infrastructure. Furthermore, BrowserStack’s Automate supports integration with popular test automation frameworks, including Puppeteer. It enables testing on 3500+ real device-OS-browser combinations. This helps organizations reduce the cost and additional effort required for infrastructure maintenance.

Conclusion

Puppeteer is a modern automation tool that allows both test automation and web scraping. It provides control over the Chrome browser by offering high-level APIs lacking in traditional test automation frameworks.

It also offers many advanced features, such as waiting for specific events rather than waiting for elements or using hard-coded timeouts. waitUntil is a special feature provided by Puppeteer that allows waiting for many events.

However, testing on real devices is crucial for modern web applications. It further enhances the accuracy of results by simulating true user behavior and accounting for various environmental factors, such as network conditions and device performance.

BrowserStack Automate enables scalable, on-demand testing on real devices, eliminating the need for complex infrastructure management while providing a comprehensive solution for quality assurance across various user scenarios.

Try BrowserStack Now