Scrape Table from Website using Python – Selenium
Selenium is the automation software testing tool that obtains the website, performs various actions, or obtains the data from the website. It was chiefly developed for easing the testing work by automating the web applications. Nowadays, apart from being used for testing, it can also be used for making tedious work interesting. Do you know that with the help of Selenium, you can also extract data from the table on the website? The answer is Yes, we can easily scrap the table data from the website. What you need to do in order to scrape table data from the website is explained in this article.
Approach to be followed:
Let us consider the simple HTML program containing tables only to understand the approach of scraping the table from the website.
Follow the below-given steps:
Once you have created the HTML file, you can follow the below steps and extract data from the table from the website on your own.
- First, declare the web driver
driver=webdriver.Chrome(executable_path=”Declare the path where web driver is installed”)
- Now, open the website from which you want to obtain table data
driver.get("Specify the path of the website")
- Next, you need to find rows in the table
rows=1+len(driver.find_elements_by_xpath("Specify the altered path"))
Here, the altered xpath means that if xpath of the row 1 is /html/body/table/tbody/tr then, altered xpath will be /html/body/table/tbody/tr What needs to be done here is to remove the index value of table row.
NOTE: Remember to add 1 to the row’s value for the table header as it was not included while calculating the table rows.
- Further, find columns in the table
cols=len(driver.find_elements_by_xpath("Specify the altered path"))
Here, the altered xpath means that if xpath of the column showing output Vinayak is /html/body/table/tbody/tr/td then, altered xpath will be /html/body/table/tbody/tr/td What needs to be done here is to remove the index value of table row and table data.
- Moreover, obtain data from each column of the table body
for r in range(2, rows+1): for p in range(1, cols+1): value = driver.find_element_by_xpath("Specify the altered path").text
Here, the altered xpath means that if xpath of the column showing output Vinayak is /html/body/table/tbody/tr/td then, altered xpath will be /html/body/table/tbody/tr[“+str(r)+”]/td[“+str(p)+”] What needs to be done here is to add the str(r) and str(p) for the index value of table row and table data respectively.
- Finally, print data of the table
print(value, end=' ') print()
How to scrape table data from the website in Selenium?
As we have now seen the approach to be followed to extract the table data while using the automation tool Selenium. Now, let’s see the complete example for the scrapping table data from the website. We will use this website to extract its table data in the given below program.
Further, run the python code using:
Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.
To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course