Python – Preprocessing of Tamil Text
Preprocessing is the major part of Natural Language Processing. In order to classify any text with high accuracy, cleaned data plays a major role. So, the first step in NLP before analyzing or classifying is preprocessing of data. Many python libraries support preprocessing for the English language. But for the Tamil Language, there are very few preprocessing libraries available. Here is an example of a few preprocessing techniques for Tamil text.
The preprocessing techniques involved in this article are
Attention reader! Don’t stop learning now. Get hold of all the important Machine Learning Concepts with the Machine Learning Foundation Course at a student-friendly price and become industry ready.
- Punctuation removal
- Stop words removal
If there are any of the above punctuations in the text, they will be removed after preprocessing. This can be removed by using the python string module.
வெற்றி பெற வேண்டும் என்ற பதற்றம் இல்லாமல் இருப்பது தான் வெற்றி பெறுவதற்கான சிறந்த வழி
Explanation: All the punctuations in the given text are removed.
Tokenization is nothing but splitting each word in a sentence into a token which will be used for further classification. To convert a text into tokens, a regular expression module in python is used.
[‘வெற்றி’, ‘பெற’, ‘வேண்டும்’, ‘என்ற’, ‘பதற்றம்’, ‘இல்லாமல்’, ‘இருப்பது’, ‘தான்’, ‘வெற்றி’, ‘பெறுவதற்கான’, ‘சிறந்த’, ‘வழி’]
Explanation: All the words in a sentence are split into tokens.
Stop words removal:
Stop words are frequently used words in a language. These words are unnecessary for the meaning of the sentence. Removal of stop words can be done by using the NLTK package in python. The NLTK package supports many languages like English, French, German, Finnish, Italian, etc but not the Tamil language. So, download the stop words for Tamil in the given link – Github Link and name the file as tamil and place it in the below location in your system – “ ….\AppData\Roaming\nltk_data\corpora\stopwords“
After this process, the NLTK package supports Tamil stop words also
[‘வெற்றி’, ‘பெற’, ‘பதற்றம்’, ‘இல்லாமல்’, ‘இருப்பது’, ‘வெற்றி’, ‘பெறுவதற்கான’, ‘சிறந்த’, ‘வழி’]
Explanation: The stop words ‘வேண்டும்’, ‘என்ற’ and ‘தான்’ are removed.