NLP | Storing Conditional Frequency Distribution in Redis

The nltk.probability.ConditionalFreqDist class is a container for FreqDist instances, with one FreqDist per condition. It is used to count frequencies that are dependent on another condition, such as another word or a class label. It is being used here to create an API-compatible class on top of Redis using the RedisHashFreqDist .
In the code given below, a RedisConditionalHashFreqDist class that extends nltk.probability.ConditionalFreqDist and overrides the __getitem__() method. Override __getitem__() so as to create an instance of RedisHashFreqDist instead of a FreqDist.

Code :

filter_none

edit
close

play_arrow

link
brightness_4
code

from nltk.probability import ConditionalFreqDist
from rediscollections import encode_key
  
class RedisConditionalHashFreqDist(ConditionalFreqDist):
    def __init__(self, r, name, cond_samples = None):
        self._r = r
        self._name = name
        ConditionalFreqDist.__init__(self, cond_samples)
          
        for key in self._r.keys(encode_key('% s:*' % name)):
            condition = key.split(':')[1]
            # calls self.__getitem__(condition)
            self[condition] 
              
    def __getitem__(self, condition):
        if condition not in self._fdists:
            key = '% s:% s' % (self._name, condition)
            val = RedisHashFreqDist(self._r, key)
            super(RedisConditionalHashFreqDist, self).__setitem__(
                    condition, val)
        return super(
                RedisConditionalHashFreqDist, self).__getitem__(condition)
    def clear(self):
        for fdist in self.values():
            fdist.clear()

chevron_right


An instance of this class can be created by passing in a Redis connection and a base name. After that, it works just like a ConditionalFreqDist as shown in the code below :
Code :

filter_none

edit
close

play_arrow

link
brightness_4
code

from redis import Redis
from redisprob import RedisConditionalHashFreqDist
  
r = Redis()
rchfd = RedisConditionalHashFreqDist(r, 'condhash')
  
print (rchfd.N())
  
print (rchfd.conditions())
  
rchfd['cond1']['foo'] += 1
  
print (rchfd.N())
  
print (rchfd['cond1']['foo'])
  
print (rchfd.conditions())
  
rchfd.clear()

chevron_right



Output :

0
[]
1
1
['cond1']

How it works ?

  • The RedisConditionalHashFreqDist uses name prefixes to reference RedisHashFreqDist instances.
  • The name passed into the RedisConditionalHashFreqDist is a base name that is combined with each condition to create a unique name for each RedisHashFreqDist.
  • For example, if the base name of the RedisConditionalHashFreqDist is ‘condhash’, and the condition is ‘cond1’, then the final name for the RedisHashFreqDist is ‘condhash:cond1’.
  • This naming pattern is used at initialization to find all the existing hash maps using the keys command.
  • By searching for all keys matching ‘condhash:*’, user can identify all the existing conditions and create an instance of RedisHashFreqDist for each.
  • Combining strings with colons is a common naming convention for Redis keys as a way to define namespaces.
  • Each RedisConditionalHashFreqDist instance defines a single namespace of hash maps.

RedisConditionalHashFreqDist also defines a clear() method. This is a helper method that calls clear() on all the internal RedisHashFreqDist instances. The clear() method is not defined in ConditionalFreqDist.



My Personal Notes arrow_drop_up

Check out this Author's contributed articles.

If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.