Skip to content
Related Articles

Related Articles

Improve Article
NLP | Training Tagger Based Chunker | Set 1
  • Last Updated : 23 Feb, 2019

To train a chunker is an alternative to manually specifying regular expression (regex) chunk patterns. But manually training to specify the expression is a tedious task to do as it follows the hit and trial method to get the exact right patterns. So, existing corpus data can be used to train chunkers.

In the codes below, we are using treebank_chunk corpus to produce chunked sentences in the form of trees.
-> To train a tagger-based chunker – chunked_sents() methods are used by a TagChunker class.
-> To extract a list of (pos, iob) tuples from a list of Trees – the TagChunker class uses a helper function, conll_tag_chunks().

These tuples are then finally used to train a tagger. and it learns IOB tags for part-of-speech tags.

Code #1 : Let’s understand the Chunker class for training.

from nltk.chunk import ChunkParserI
from nltk.chunk.util import tree2conlltags, conlltags2tree
from nltk.tag import UnigramTagger, BigramTagger
from tag_util import backoff_tagger
def conll_tag_chunks(chunk_data):
    tagged_data = [tree2conlltags(tree) for 
                    tree in chunk_data]
    return [[(t, c) for (w, t, c) in sent] 
            for sent in tagged_data]
class TagChunker(ChunkParserI):
    def __init__(self, train_chunks, 
                 tagger_classes =[UnigramTagger, BigramTagger]):
        train_data = conll_tag_chunks(train_chunks)
        self.tagger = backoff_tagger(train_data, tagger_classes)
    def parse(self, tagged_sent):
        if not tagged_sent: 
            return None
        (words, tags) = zip(*tagged_sent)
        chunks = self.tagger.tag(tags)
        wtc = zip(words, chunks)
        return conlltags2tree([(w, t, c) for (w, (t, c)) in wtc])

Output :

Training TagChunker

Code #2 : Using the Tag Chunker.

# loading libraries
from chunkers import TagChunker
from nltk.corpus import treebank_chunk
# data from treebank_chunk corpus
train_data = treebank_chunk.chunked_sents()[:3000]
test_data = treebank_chunk.chunked_sents()[3000:]
# Initailazing 
chunker = TagChunker(train_data)

Code #3 : Evaluating the TagChunker

# testing
score = chunker.evaluate(test_data)
a = score.accuracy()
p = score.precision()
r = recall
print ("Accuracy of TagChunker : ", a)
print ("\nPrecision of TagChunker : ", p)
print ("\nRecall of TagChunker : ", r)

Output :

Accuracy of TagChunker : 0.9732039335251428

Precision of TagChunker : 0.9166534370535006

Recall of TagChunker : 0.9465573770491803

 Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.  

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course. And to begin with your Machine Learning Journey, join the Machine Learning – Basic Level Course

My Personal Notes arrow_drop_up
Recommended Articles
Page :