Python NLTK | nltk.tokenizer.word_tokenize()
With the help of
nltk.tokenize.word_tokenize() method, we are able to extract the tokens from string of characters by using
tokenize.word_tokenize() method. It actually returns the syllables from a single word. A single word can contain one or two syllables.
Return : Return the list of syllables of words.
Example #1 :
In this example we can see that by using
tokenize.word_tokenize() method, we are able to extract the syllables from stream of words or sentences.
[‘An’, ‘ti’, ‘dis’, ‘es’, ‘ta’, ‘blish’, ‘men’, ‘ta’, ‘ria’, ‘nism’]
Example #2 :
[‘Ga’, ‘me’, ‘to’, ‘phy’, ‘te’]
Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.
To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course