Open In App

Python NLTK | nltk.tokenizer.word_tokenize()

Improve
Improve
Like Article
Like
Save
Share
Report

With the help of nltk.tokenize.word_tokenize() method, we are able to extract the tokens from string of characters by using tokenize.word_tokenize() method. It actually returns the syllables from a single word. A single word can contain one or two syllables.

Syntax : tokenize.word_tokenize()
Return : Return the list of syllables of words.

Example #1 :
In this example we can see that by using tokenize.word_tokenize() method, we are able to extract the syllables from stream of words or sentences.




# import SyllableTokenizer() method from nltk
from nltk import word_tokenize
     
# Create a reference variable for Class word_tokenize
tk = SyllableTokenizer()
     
# Create a string input
gfg = "Antidisestablishmentarianism"
     
# Use tokenize method
geek = tk.tokenize(gfg)
     
print(geek)


Output :

[‘An’, ‘ti’, ‘dis’, ‘es’, ‘ta’, ‘blish’, ‘men’, ‘ta’, ‘ria’, ‘nism’]

Example #2 :




# import SyllableTokenizer() method from nltk
from nltk.tokenize import word_tokenize
     
# Create a reference variable for Class word_tokenize
tk = SyllableTokenizer()
     
# Create a string input
gfg = "Gametophyte"
     
# Use tokenize method
geek = tk.tokenize(gfg)
     
print(geek)


Output :

[‘Ga’, ‘me’, ‘to’, ‘phy’, ‘te’]


Last Updated : 12 Jun, 2019
Like Article
Save Article
Previous
Next
Share your thoughts in the comments
Similar Reads