Python NLTK | nltk.tokenizer.word_tokenize()
With the help of nltk.tokenize.word_tokenize()
method, we are able to extract the tokens from string of characters by using tokenize.word_tokenize()
method. It actually returns the syllables from a single word. A single word can contain one or two syllables.
Syntax : tokenize.word_tokenize()
Return : Return the list of syllables of words.
Example #1 :
In this example we can see that by using tokenize.word_tokenize()
method, we are able to extract the syllables from stream of words or sentences.
from nltk import word_tokenize
tk = SyllableTokenizer()
gfg = "Antidisestablishmentarianism"
geek = tk.tokenize(gfg)
print (geek)
|
Output :
[‘An’, ‘ti’, ‘dis’, ‘es’, ‘ta’, ‘blish’, ‘men’, ‘ta’, ‘ria’, ‘nism’]
Example #2 :
from nltk.tokenize import word_tokenize
tk = SyllableTokenizer()
gfg = "Gametophyte"
geek = tk.tokenize(gfg)
print (geek)
|
Output :
[‘Ga’, ‘me’, ‘to’, ‘phy’, ‘te’]
Last Updated :
12 Jun, 2019
Like Article
Save Article
Share your thoughts in the comments
Please Login to comment...