With the help of
nltk.tokenize.word_tokenize() method, we are able to extract the tokens from string of characters by using
tokenize.word_tokenize() method. It actually returns the syllables from a single word. A single word can contain one or two syllables.
Return : Return the list of syllables of words.
Example #1 :
In this example we can see that by using
tokenize.word_tokenize() method, we are able to extract the syllables from stream of words or sentences.
[‘An’, ‘ti’, ‘dis’, ‘es’, ‘ta’, ‘blish’, ‘men’, ‘ta’, ‘ria’, ‘nism’]
Example #2 :
[‘Ga’, ‘me’, ‘to’, ‘phy’, ‘te’]
- Python NLTK | nltk.tokenize.TabTokenizer()
- Python NLTK | nltk.tokenize.SpaceTokenizer()
- Python NLTK | nltk.tokenize.StanfordTokenizer()
- Python NLTK | nltk.TweetTokenizer()
- Python NLTK | nltk.tokenize.mwe()
- Python NLTK | nltk.WhitespaceTokenizer
- Python NLTK | nltk.tokenize.LineTokenizer
- Python NLTK | nltk.tokenize.SExprTokenizer()
- Python | NLTK nltk.tokenize.ConditionalFreqDist()
- Python | Gender Identification by name using NLTK
- Python NLTK | tokenize.regexp()
- Python NLTK | tokenize.WordPunctTokenizer()
- Creating a Basic hardcoded ChatBot using Python-NLTK
- Python | Measure similarity between two sentences using cosine similarity
- Text Preprocessing in Python | Set - 1
- Text Preprocessing in Python | Set 2
- Text detection using Python
- Python | Text Summarizer
If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to email@example.com. See your article appearing on the GeeksforGeeks main page and help other Geeks.
Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.