Skip to content
Related Articles

Related Articles

Python NLTK | nltk.tokenizer.word_tokenize()

View Discussion
Improve Article
Save Article
  • Difficulty Level : Easy
  • Last Updated : 12 Jun, 2019
View Discussion
Improve Article
Save Article

With the help of nltk.tokenize.word_tokenize() method, we are able to extract the tokens from string of characters by using tokenize.word_tokenize() method. It actually returns the syllables from a single word. A single word can contain one or two syllables.

Syntax : tokenize.word_tokenize()
Return : Return the list of syllables of words.

Example #1 :
In this example we can see that by using tokenize.word_tokenize() method, we are able to extract the syllables from stream of words or sentences.




# import SyllableTokenizer() method from nltk
from nltk import word_tokenize
     
# Create a reference variable for Class word_tokenize
tk = SyllableTokenizer()
     
# Create a string input
gfg = "Antidisestablishmentarianism"
     
# Use tokenize method
geek = tk.tokenize(gfg)
     
print(geek)

Output :

[‘An’, ‘ti’, ‘dis’, ‘es’, ‘ta’, ‘blish’, ‘men’, ‘ta’, ‘ria’, ‘nism’]

Example #2 :




# import SyllableTokenizer() method from nltk
from nltk.tokenize import word_tokenize
     
# Create a reference variable for Class word_tokenize
tk = SyllableTokenizer()
     
# Create a string input
gfg = "Gametophyte"
     
# Use tokenize method
geek = tk.tokenize(gfg)
     
print(geek)

Output :

[‘Ga’, ‘me’, ‘to’, ‘phy’, ‘te’]

My Personal Notes arrow_drop_up
Recommended Articles
Page :

Start Your Coding Journey Now!