Skip to content
Related Articles

Related Articles

NLP | Distributed chunking with Execnet

View Discussion
Improve Article
Save Article
  • Last Updated : 12 Jun, 2019
View Discussion
Improve Article
Save Article

The article aims to perform chunking and tagging over an execnet gateway. Here two objects will be sent instead of one, and a Tree is received, which requires pickling and unpickling for serialization.

How it works ?

  • Use a pickled tagger.
  • First, pickle the default chunker used by nltk.chunk.ne_chunk(), though any chunker would do.
  • Next, make a gateway for the remote_chunk module, get a channel, and send the pickled tagger and chunker over.
  • Then, receive a pickled Tree, which can be unpickled and inspected to see the result. Finally, exit the gateway:

Code : Explaining the working

# importing libraries
import execnet, remote_chunk
import, nltk.tag, nltk.chunk
import pickle
from nltk.corpus import treebank_chunk
tagger = pickle.dumps(
chunker = pickle.dumps(
gw = execnet.makegateway()
channel = gw.remote_exec(remote_chunk)
chunk_tree = pickle.loads(channel.receive())
print (chunk_tree)

Output :

Tree('S', [Tree('PERSON', [('Pierre', 'NNP')]), Tree('ORGANIZATION',
[('Vinken', 'NNP')]), (', ', ', '), ('61', 'CD'), ('years', 'NNS'),
('old', 'JJ'), (', ', ', '), ('will', 'MD'), ('join', 'VB'), ('the',
'DT'), ('board', 'NN'), ('as', 'IN'), ('a', 'DT'), ('nonexecutive',
'JJ'), ('director', 'NN'), ('Nov.', 'NNP'), ('29', 'CD'), ('.', '.')]) 

Communication this time is slightly different as in figure given below –

  • The module is just a little bit more complicated than the module.
  • In addition to receiving a pickled tagger, it also expects to receive a pickled chunker that implements the ChunkerIinterface
  • Once it has both a tagger and a chunker, it expects to receive any number of tokenized sentences, which it tags and parses into a Tree. This Tree is then pickled and sent back over the channel:

Code : Explaining the above working

import pickle
if __name__ == '__channelexec__':
    tagger = pickle.loads(channel.receive())
    chunker = pickle.loads(channel.receive())
for sentence in channel:
    chunk_tree = chunker.parse(tagger.tag(sent))

remote_chunk module’s only external dependency is the pickle module, which is part of the Python standard library. It doesn’t need to import any NLTK modules in order to use the tagger or chunker, because all the necessary data is pickled and sent over the channel.

My Personal Notes arrow_drop_up
Recommended Articles
Page :

Start Your Coding Journey Now!