NLP | Training Tagger Based Chunker | Set 1

To train a chunker is an alternative to manually specifying regular expression (regex) chunk patterns. But manually training to specify the expression is a tedious task to do as it follows the hit and trial method to get the exact right patterns. So, existing corpus data can be used to train chunkers.

In the codes below, we are using treebank_chunk corpus to produce chunked sentences in the form of trees.
-> To train a tagger-based chunker – chunked_sents() methods are used by a TagChunker class.
-> To extract a list of (pos, iob) tuples from a list of Trees – the TagChunker class uses a helper function, conll_tag_chunks().

These tuples are then finally used to train a tagger. and it learns IOB tags for part-of-speech tags.

Code #1 : Let’s understand the Chunker class for training.

from nltk.chunk import ChunkParserI
from nltk.chunk.util import tree2conlltags, conlltags2tree
from nltk.tag import UnigramTagger, BigramTagger
from tag_util import backoff_tagger


def conll_tag_chunks(chunk_data):

    tagged_data = [tree2conlltags(tree) for
                    tree in chunk_data]

    return [[(t, c) for (w, t, c) in sent]
            for sent in tagged_data]

class TagChunker(ChunkParserI):

    def __init__(self, train_chunks,
                 tagger_classes =[UnigramTagger, BigramTagger]):

        train_data = conll_tag_chunks(train_chunks)
        self.tagger = backoff_tagger(train_data, tagger_classes)

    def parse(self, tagged_sent):
        if not tagged_sent:
            return None

        (words, tags) = zip(*tagged_sent)
        chunks = self.tagger.tag(tags)
        wtc = zip(words, chunks)

        return conlltags2tree([(w, t, c) for (w, (t, c)) in wtc])

Output :

Training TagChunker

Code #2 : Using the Tag Chunker.

# loading libraries
from chunkers import TagChunker
from nltk.corpus import treebank_chunk

# data from treebank_chunk corpus
train_data = treebank_chunk.chunked_sents()[:3000]
test_data = treebank_chunk.chunked_sents()[3000:]

# Initailazing
chunker = TagChunker(train_data)

Code #3 : Evaluating the TagChunker

# testing
score = chunker.evaluate(test_data)

a = score.accuracy()
p = score.precision()
r = recall

print ("Accuracy of TagChunker : ", a)
print ("\nPrecision of TagChunker : ", p)
print ("\nRecall of TagChunker : ", r)

Output :

Accuracy of TagChunker : 0.9732039335251428

Precision of TagChunker : 0.9166534370535006

Recall of TagChunker : 0.9465573770491803