Skip to content
Related Articles

Related Articles

Implementing Web Crawler using Abstract Factory Design Pattern in Python

View Discussion
Improve Article
Save Article
  • Difficulty Level : Medium
  • Last Updated : 11 Oct, 2021

In the Abstract Factory design pattern, every product has an abstract product interface. This approach facilitates the creation of families of related objects that is independent of their factory classes. As a result, you can change the factory at runtime to get a different object  – simplifies the replacement of the product families.   

In this design pattern, the client uses an abstract factory interface to access objects. The abstract interface separates the creation of objects from the client, which makes the manipulation easier and isolates the concrete classes from the client. However, adding new products to the existing factory is difficult because you need to extend the factory interface, which includes changing the abstract factory interface class and all its subclasses.  

Let’s look into the web crawler implementation in Python for a better understanding. As shown in the following diagram, you have an abstract factory interface class – AbstractFactory – and two concrete factory classes – HTTPConcreteFactory and FTPConcreteFactory. These two concrete classes are derived from the AbstractFactory class and have methods to create instances of three interfaces – ProtocolAbstractProduct, PortAbstractProduct, and CrawlerAbstractProduct.

Since AbstractFactory class acts as an interface for the factories such as HTTPConcreteFactory and FTPConcreteFactory, it has three abstract methods – create_protocol(), create_port(), create_crawler(). These methods are redefined in the factory classes. That means HTTPConcreteFactory class creates its family of related objects such as HTTPPort, HTTPSecurePort, and HTTPSecureProtocol, whereas, FTPConcreteFactory class creates FTPPort, FTPProtocol, and FTPCrawler.


import abc
import urllib
import urllib.error
import urllib.request
from bs4 import BeautifulSoup
class AbstractFactory(object, metaclass=abc.ABCMeta):
    """ Abstract Factory Interface """
    def __init__(self, is_secure):
        self.is_secure = is_secure
    def create_protocol(self):
    def create_port(self):
    def create_crawler(self):
class HTTPConcreteFactory(AbstractFactory):
    """ Concrete Factory for building HTTP connection. """
    def create_protocol(self):
        if self.is_secure:
            return HTTPSecureProtocol()
        return HTTPProtocol()
    def create_port(self):
        if self.is_secure:
            return HTTPSecurePort()
        return HTTPPort()
    def create_crawler(self):
        return HTTPCrawler()
class FTPConcreteFactory(AbstractFactory):
    """ Concrete Factory for building FTP connection """
    def create_protocol(self):
        return FTPProtocol()
    def create_port(self):
        return FTPPort()
    def create_crawler(self):
        return FTPCrawler()
class ProtocolAbstractProduct(object, metaclass=abc.ABCMeta):
    """ An abstract product, represents protocol to connect """
    def __str__(self):
class HTTPProtocol(ProtocolAbstractProduct):
    """ An concrete product, represents http protocol """
    def __str__(self):
        return 'http'
class HTTPSecureProtocol(ProtocolAbstractProduct):
    """ An concrete product, represents https protocol """
    def __str__(self):
        return 'https'
class FTPProtocol(ProtocolAbstractProduct):
    """ An concrete product, represents ftp protocol """
    def __str__(self):
        return 'ftp'
class PortAbstractProduct(object, metaclass=abc.ABCMeta):
    """ An abstract product, represents port to connect """
    def __str__(self):
class HTTPPort(PortAbstractProduct):
    """ A concrete product which represents http port. """
    def __str__(self):
        return '80'
class HTTPSecurePort(PortAbstractProduct):
    """ A concrete product which represents https port """
    def __str__(self):
        return '443'
class FTPPort(PortAbstractProduct):
    """ A concrete products which represents ftp port. """
    def __str__(self):
        return '21'
class CrawlerAbstractProduct(object, metaclass=abc.ABCMeta):
    """ An Abstract product, represents parser to parse web content """
    def __call__(self, content):
class HTTPCrawler(CrawlerAbstractProduct):
    def __call__(self, content):
        """ Parses web content """
        filenames = []
        soup = BeautifulSoup(content, "html.parser")
        links = soup.table.findAll('a')
        for link in links:
        return '\n'.join(filenames)
class FTPCrawler(CrawlerAbstractProduct):
    def __call__(self, content):
        """ Parse Web Content """
        content = str(content, 'utf-8')
        lines = content.split('\n')
        filenames = []
        for line in lines:
            splitted_line = line.split(None, 8)
            if len(splitted_line) == 9:
        return '\n'.join(filenames)
class Connector(object):
    """ A client """
    def __init__(self, abstractfactory):
        """ calling all attributes
of a connector according to abstractfactory class. """
        self.protocol = abstractfactory.create_protocol()
        self.port = abstractfactory.create_port()
        self.crawl = abstractfactory.create_crawler()
    def read(self, host, path):
        url = str(self.protocol) + '://' + host + ':' + str(self.port) + path
        print('Connecting to', url)
        return urllib.request.urlopen(url, timeout=10).read()
if __name__ == "__main__":
    con_domain = ''
    con_path = '/pub/FreeBSD/'
    con_protocol = input('Choose the protocol \
                    (0-http, 1-ftp): ')
    if con_protocol == '0':
        is_secure = input('Use secure connection? (1-yes, 0-no):')
        if is_secure == '1':
            is_secure = True
            is_secure = False
        abstractfactory = HTTPConcreteFactory(is_secure)
        is_secure = False
        abstractfactory = FTPConcreteFactory(is_secure)
    connector = Connector(abstractfactory)
        data =, con_path)
    except urllib.error.URLError as e:
        print('Cannot access resource with this method', e)



The goal of the program is to crawl the website using the HTTP protocol or FTP protocol. Here, we need to consider three scenarios while implementing the code. 

  1. Protocol
  2. Port
  3. Crawler

These three scenarios differ in the HTTP and FTP web access models. So, here we need to create two factories, one for creating HTTP products and another for creating FTP products – HTTPConcreteFactory and FTPConcreteFactory. These two concrete factories are derived from an abstract factory – AbstractFactory

An abstract interface is used because the operation methods are the same for both factory classes, only the implementation is different, and hence the client code can determine which factory to using during the runtime. Let’s analyze the products created by each factory. 

In the case of protocol product, HTTP concrete factory creates either http or https protocol, whereas, FTP concrete factory creates ftp protocol. For port products, HTTP concrete factory generates either 80 or 443 as a port product, and the FTP factory generates 21 as a port product. And finally, the crawler implementation differs because the website structure is different for HTTP and FTP.  

Here, the created object has the same interface, whereas the created concrete objects are different for every factory. Say, for example, the port products such as HTTP port, HTTP Secure port, and FTP port have the same interface, but the concrete objects for both factories are different. The same is applicable for protocol and crawler as well. 

Finally, the connector class accepts a factory and uses this factory to inject all attributes of the connector based on the factory class.

My Personal Notes arrow_drop_up
Recommended Articles
Page :

Start Your Coding Journey Now!