Waiting for page to load in RSelenium in R

In this article, we will discuss how to wait for a page to load in RSelenium. This is a very important functionality that we have to include in our scrapping codes so, that the page gets loaded completely before we scrape the desired part of the code.

What is RSelenium?

RSelenium is an R Programming Langauge package that provides developers with a way to automate web tasks, parse web pages, fill out forms, etc. It supports all your favorite browsers like Chrome, Microsoft Edge, Bing, Firefox, and many more and it is supported by a wide range of OS such as Windows, Mac, and Linux. It is very popular in its niche as it is simple to use yet very powerful.

What problem are we facing?

The problem we are facing is that whenever a page loads we immediately try to look for an element let’s say an input field for a discussion matter. But when a page loads it takes some time for the input field to be clickable but we still look for it, which throws an error and stops your script as the code was looking for something which did not exist (yet).

Take a look at this diagram for a little more detail. In this example, we are not waiting for the page to finish loading and immediately look for the input field.

Sys.Sleep()

The Sys.sleep() stops the further code from being executed. It takes one argument which is the amount of seconds you want the code to sleep for.

Sys.sleep(5)

This code will sleep for 5 seconds and then execute the further code. We can insert this into our code to make it look something like this.

R

# Import the library
library(RSelenium)

# Intializing the rsDriver with google chrome
rD <- rsDriver(browser = "chrome",
               chromever = "113.0.5672.63",
               port = 4445L)

# Opening browser
remDr <- rD$client
remDr$open()

# Making the code sleep for 5 seconds
Sys.sleep(5)

# Opening GeeksForGeeks
remDr$navigate("https://www.geeksforgeeks.org/")

# Perform any action you want here

But this is bad design because sometimes the browser may take longer or a lesser time to load. We can fix this above problem by setting Timeouts in RSelenium.

Timeouts

The below code will wait for 20 seconds to let the page load. If the page fails to load within 20 seconds it will throw an error. The beauty of this solution is that it won’t wait for the entire 20 seconds if the page loads earlier. For example: if the page loads in 7 seconds it will just perform other actions without waiting for another 14 seconds.

R

remDr$setTimeout(type = "page load",
                 milliseconds = 20000)

Different types of Timeouts

• Implicit: This wait is applied to every element you try to find. It will wait for a certain amount of milliseconds before saying that the element was not found.
• Page Load: As we just used it waits for the given period of time to let the page load.
• Script: This type of timeout is used to wait a certain period of time to let a “script” run. Note that script here refers to a JavaScript code execution.
• Explicit: This type of wait, waits until a given condition is true, The condition can be something along the lines of “wait until an element is clickable”. This is useful if you don’t want to wait for the entire page to load and only an element.

Conclusion

We can use these techniques to empower our skills and automate tasks without having to worry about them not being fully loaded. Here is the full code if you’re still confused.

R

# Import the library
library(RSelenium)

# Intializing the rsDriver with google chrome
rD <- rsDriver(browser = "chrome",
               chromever = "113.0.5672.63",
               port = 4445L)

# Opening browser
remDr <- rD$client
remDr$open()

# Setting the timeout for page to
# load for 20 seconds
remDr$setTimeout(type = "page load",
                 milliseconds = 20000)

# Opening GeeksForGeeks
remDr$navigate("https://www.geeksforgeeks.org/")

# Perform any action you want here