HTML Cleaning and Entity Conversion | Python

The very important and always ignored task on web is the cleaning of text. Whenever one thinks to parse HTML, embedded Javascript and CSS is always avoided. The users are only interested in tags and text present on the webserver.

lxml installation –
It is a Python binding for C libraries – libxslt and libxml2. So maintaining a Python base, it is very fast HTML parsing and XML library. To let it work – C libraries also need to be installed. The link – http://lxml.de/installation.html will provide all the installation instructions.

sudo apt-get install python-lxml or
pip install lxml

Cleaning task is performed using clean_html() function present in the lxml.html.clean module. This function removes the unnecessary HTML tags. In the code below, clean_html() function in the lxml.html.clean module is used to remove unnecessary HTML tags and embedded JavaScript from an HTML string.

Code – Cleaning of the text

filter_none

edit
close

play_arrow

link
brightness_4
code

import lxml.html.clean
lxml.html.clean.clean_html('<html><head></head>
                           <bodyonload = loadfunc()>my text</body></html>')

chevron_right


Output :

'<div><body>my text</body></div>'

As you can see that the results are much easier and cleaner. Thus, makes our job easy to deal with the HTML.

The lxml.html.clean_html() function iterates over the string as it parses the HTML string into a tree. It then removes all nodes that don’t hold much importance. Using embedded JavaScript, the function also cleans nodes of unnecessary attributes like embedded JavaScript using regex (regular expression) substitution and matching. This function defines a default Cleaner class that’s used clean_html() method is called. By creating self instance, the class behavior can be customized.

Converting HTML Entities –

Strings such as “&” or “<” are HTML entities. These are normal ASCII character encoding having special uses in HTML. “<” is the entity for “<" because "<" is present within HTML tags and it is the beginning character for an HTML tag. So, to escape it "<" entity is defined. "&" is entity code for "&".
To process the text within an HTML document, convert these entities back to their normal characters so as to recognize them and use them appropriately.

Requirement :
1) install BeautifulSoup
2) sudo easy_install beautifulsoup4 or sudo pip install beautifulsoup4

It is an HTML parser library used for entity conversion. It simply creates an instance of BeautifulSoup given a string containing HTML entities. And then it retrieves the string attribute:

Code –

filter_none

edit
close

play_arrow

link
brightness_4
code

# importing BeautifulSoup
from bs4 import BeautifulSoup
  
print (BeautifulSoup('<').string)
  
print (BeautifulSoup('&').string)

chevron_right


Output :

'<'
'&'

But the reverse for it is not possible i.e. for ‘<' in BeautifulSoup, a None result is obtained as it is invalid in HTML. BeautifulSoup looks for tokens that look similar to an entity and in order to convert the HTML entities, it replaces them with their corresponding value in the htmlentitydefs.name2codepoint dictionary which is there in the python standard library.



My Personal Notes arrow_drop_up

Check out this Author's contributed articles.

If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.




Article Tags :

Be the First to upvote.


Please write to us at contribute@geeksforgeeks.org to report any issue with the above content.