Get contents of entire page using Selenium
In this article, we will discuss ways to get the contents of the entire page using Selenium. There can broadly be two methods for the same. Let’s discuss them in detail.
Method 1:
For extracting the visible text from the entire page, we can use the find_element_by_* methods which help us find or locate the elements on the page. Then, We will use the text method which helps to retrieve the text from a specific web element.
Approach
- Import module
- Instantiate driver
- Get content of the page
- Display contents scraped
- Close driver
Syntax:
driver.find_element_by_class_xpath(“/html/body”).text
To find or locate multiple elements on a page:
- find_element_by_link_text
- find_element_by_partial_link_text
- find_element_by_xpath
- find_element_by_tag_name
- find_element_by_class_name
- find_element_by_css_selector
- find_element_by_id
- find_element_by_name
We can use these above methods for finding or locating elements on a entire page. Most commonly used method is find_element_by_xpath which helps us to easily locate any elements. We will use appropriate methods as per our requirement.
Program:
Python3
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(
print (driver.find_element_by_xpath( "/html/body" ).text)
driver.close()
|
Output:
Method 2:
There is one another method available for achieving our desired output. This one line will retrieve the entire text of the web page. Once we get the extracted data, with the help of file system, we will store the result inside the result.html file.
Approach:
- Import module
- Instantiate webdriver
- Get contents from the URL
- Open a file
- Save contents to a file
- Close file
- Close driver
Syntax:
driver.page_source
Program:
Python3
from selenium import webdriver
from webdriver_manager.chrome import ChromeDriverManager
driver = webdriver.Chrome(ChromeDriverManager().install())
driver.get(
page = driver.page_source.encode( 'utf-8' )
file_ = open ( 'result.html' , 'wb' )
file_.write(page)
file_.close()
driver.close()
|
Output:
Click here to download the output file of above program.
Last Updated :
25 Feb, 2021
Like Article
Save Article
Share your thoughts in the comments
Please Login to comment...