Open In App
Related Articles

NLP | Training Tagger Based Chunker | Set 1

Like Article
Save Article
Report issue
To train a chunker is an alternative to manually specifying regular expression (regex) chunk patterns. But manually training to specify the expression is a tedious task to do as it follows the hit and trial method to get the exact right patterns. So, existing corpus data can be used to train chunkers. In the codes below, we are using treebank_chunk corpus to produce chunked sentences in the form of trees. -> To train a tagger-based chunker – chunked_sents() methods are used by a TagChunker class. -> To extract a list of (pos, iob) tuples from a list of Trees – the TagChunker class uses a helper function, conll_tag_chunks(). These tuples are then finally used to train a tagger. and it learns IOB tags for part-of-speech tags. Code #1 : Let’s understand the Chunker class for training.
from nltk.chunk import ChunkParserI
from nltk.chunk.util import tree2conlltags, conlltags2tree
from nltk.tag import UnigramTagger, BigramTagger
from tag_util import backoff_tagger
def conll_tag_chunks(chunk_data):
    tagged_data = [tree2conlltags(tree) for 
                    tree in chunk_data]
    return [[(t, c) for (w, t, c) in sent] 
            for sent in tagged_data]
class TagChunker(ChunkParserI):
    def __init__(self, train_chunks, 
                 tagger_classes =[UnigramTagger, BigramTagger]):
        train_data = conll_tag_chunks(train_chunks)
        self.tagger = backoff_tagger(train_data, tagger_classes)
    def parse(self, tagged_sent):
        if not tagged_sent: 
            return None
        (words, tags) = zip(*tagged_sent)
        chunks = self.tagger.tag(tags)
        wtc = zip(words, chunks)
        return conlltags2tree([(w, t, c) for (w, (t, c)) in wtc])

Output :
Training TagChunker
  Code #2 : Using the Tag Chunker.
# loading libraries
from chunkers import TagChunker
from nltk.corpus import treebank_chunk
# data from treebank_chunk corpus
train_data = treebank_chunk.chunked_sents()[:3000]
test_data = treebank_chunk.chunked_sents()[3000:]
# Initailazing 
chunker = TagChunker(train_data)

  Code #3 : Evaluating the TagChunker
# testing
score = chunker.evaluate(test_data)
a = score.accuracy()
p = score.precision()
r = recall
print ("Accuracy of TagChunker : ", a)
print ("\nPrecision of TagChunker : ", p)
print ("\nRecall of TagChunker : ", r)

Output :
Accuracy of TagChunker : 0.9732039335251428

Precision of TagChunker : 0.9166534370535006

Recall of TagChunker : 0.9465573770491803

Last Updated : 23 Feb, 2019
Like Article
Save Article
Share your thoughts in the comments
Similar Reads