What is Apache ZooKeeper?

Zookeeper is a distributed, open-source coordination service for distributed applications. It exposes a simple set of primitives to implement higher-level services for synchronization, configuration maintenance, and group and naming. 

In a distributed system, there are multiple nodes or machines that need to communicate with each other and coordinate their actions. ZooKeeper provides a way to ensure that these nodes are aware of each other and can coordinate their actions. It does this by maintaining a hierarchical tree of data nodes called “Znodes“, which can be used to store and retrieve data and maintain state information. ZooKeeper provides a set of primitives, such as locks, barriers, and queues, that can be used to coordinate the actions of nodes in a distributed system. It also provides features such as leader election, failover, and recovery, which can help ensure that the system is resilient to failures. ZooKeeper is widely used in distributed systems such as Hadoop, Kafka, and HBase, and it has become an essential component of many distributed applications.

Why do we need it?

• Coordination services: The integration/communication of services in a distributed environment.
• Coordination services are complex to get right. They are especially prone to errors such as race conditions and deadlock.
• Race condition-Two or more systems trying to perform some task.
• Deadlocks– Two or more operations are waiting for each other.
• To make the coordination between distributed environments easy, developers came up with an idea called zookeeper so that they don’t have to relieve distributed applications of the responsibility of implementing coordination services from scratch.

What is distributed system?

• Multiple computer systems working on a single problem.
• It is a network that consists of autonomous computers that are connected using distributed middleware.
• Key Features: Concurrent, resource sharing, independent, global, greater fault tolerance, and price/performance ratio is much better.
• Key Goals: Transparency, Reliability, Performance, Scalability.
• Challenges: Security, Fault, Coordination, and resource sharing.

Coordination Challenge

• Why is coordination in a distributed system the hard problem?
• Coordination or configuration management for a distributed application that has many systems.
• Master Node where the cluster data is stored.
• Worker nodes or slave nodes get the data from this master node. 
• single point of failure.
• synchronization is not easy.
• Careful design and implementation are needed.

Apache Zookeeper

Apache Zookeeper is a distributed, open-source coordination service for distributed systems. It provides a central place for distributed applications to store data, communicate with one another, and coordinate activities. Zookeeper is used in distributed systems to coordinate distributed processes and services. It provides a simple, tree-structured data model, a simple API, and a distributed protocol to ensure data consistency and availability. Zookeeper is designed to be highly reliable and fault-tolerant, and it can handle high levels of read and write throughput.

Zookeeper is implemented in Java and is widely used in distributed systems, particularly in the Hadoop ecosystem. It is an Apache Software Foundation project and is released under the Apache License 2.0.

Architecture of Zookeeper

Zookeeper Services

The ZooKeeper architecture consists of a hierarchy of nodes called znodes, organized in a tree-like structure. Each znode can store data and has a set of permissions that control access to the znode. The znodes are organized in a hierarchical namespace, similar to a file system. At the root of the hierarchy is the root znode, and all other znodes are children of the root znode. The hierarchy is similar to a file system hierarchy, where each znode can have children and grandchildren, and so on.

Important Components in Zookeeper

ZooKeeper Services

• Leader & Follower
• Request Processor – Active in Leader Node and is responsible for processing write requests. After processing, it sends changes to the follower nodes
• Atomic Broadcast – Present in both Leader Node and Follower Nodes. It is responsible for sending the changes to other Nodes.
• In-memory Databases (Replicated Databases)-It is responsible for storing the data in the zookeeper. Every node contains its own databases. Data is also written to the file system providing recoverability in case of any problems with the cluster.

Other Components

• Client – One of the nodes in our distributed application cluster. Access information from the server. Every client sends a message to the server to let the server know that client is alive.
• Server– Provides all the services to the client. Gives acknowledgment to the client.
• Ensemble– Group of Zookeeper servers. The minimum number of nodes that are required to form an ensemble is 3.

Zookeeper Data Model

ZooKeeper data model

In Zookeeper, data is stored in a hierarchical namespace, similar to a file system. Each node in the namespace is called a Znode, and it can store data and have children. Znodes are similar to files and directories in a file system. Zookeeper provides a simple API for creating, reading, writing, and deleting Znodes. It also provides mechanisms for detecting changes to the data stored in Znodes, such as watches and triggers. Znodes maintain a stat structure that includes: Version number, ACL, Timestamp, Data Length

Types of Znodes:

• Persistence: Alive until they’re explicitly deleted.
• Ephemeral: Active until the client connection is alive.
• Sequential: Either persistent or ephemeral.

Why do we need ZooKeeper in the Hadoop?

Zookeeper is used to manage and coordinate the nodes in a Hadoop cluster, including the NameNode, DataNode, and ResourceManager. In a Hadoop cluster, Zookeeper helps to:

• Maintain configuration information: Zookeeper stores the configuration information for the Hadoop cluster, including the location of the NameNode, DataNode, and ResourceManager.
• Manage the state of the cluster: Zookeeper tracks the state of the nodes in the Hadoop cluster and can be used to detect when a node has failed or become unavailable.
• Coordinate distributed processes: Zookeeper can be used to coordinate distributed processes, such as job scheduling and resource allocation, across the nodes in a Hadoop cluster.

Zookeeper helps to ensure the availability and reliability of a Hadoop cluster by providing a central coordination service for the nodes in the cluster.

How ZooKeeper in Hadoop Works?

ZooKeeper operates as a distributed file system and exposes a simple set of APIs that enable clients to read and write data to the file system. It stores its data in a tree-like structure called a znode, which can be thought of as a file or a directory in a traditional file system. ZooKeeper uses a consensus algorithm to ensure that all of its servers have a consistent view of the data stored in the Znodes. This means that if a client writes data to a znode, that data will be replicated to all of the other servers in the ZooKeeper ensemble.

One important feature of ZooKeeper is its ability to support the notion of a “watch.” A watch allows a client to register for notifications when the data stored in a znode changes. This can be useful for monitoring changes to the data stored in ZooKeeper and reacting to those changes in a distributed system.

In Hadoop, ZooKeeper is used for a variety of purposes, including:

• Storing configuration information: ZooKeeper is used to store configuration information that is shared by multiple Hadoop components. For example, it might be used to store the locations of NameNodes in a Hadoop cluster or the addresses of JobTracker nodes.
• Providing distributed synchronization: ZooKeeper is used to coordinate the activities of various Hadoop components and ensure that they are working together in a consistent manner. For example, it might be used to ensure that only one NameNode is active at a time in a Hadoop cluster.
• Maintaining naming: ZooKeeper is used to maintain a centralized naming service for Hadoop components. This can be useful for identifying and locating resources in a distributed system.

ZooKeeper is an essential component of Hadoop and plays a crucial role in coordinating the activity of its various subcomponents.

Reading and Writing in Apache Zookeeper

ZooKeeper provides a simple and reliable interface for reading and writing data. The data is stored in a hierarchical namespace, similar to a file system, with nodes called znodes. Each znode can store data and have children znodes. ZooKeeper clients can read and write data to these znodes by using the getData() and setData() methods, respectively. Here is an example of reading and writing data using the ZooKeeper Java API:

Java

// Connect to the ZooKeeper ensemble
ZooKeeper zk = new ZooKeeper("localhost:2181", 3000, null);

// Write data to the znode "/myZnode"
String path = "/myZnode";
String data = "hello world";
zk.create(path, data.getBytes(), Ids.OPEN_ACL_UNSAFE, CreateMode.PERSISTENT);

// Read data from the znode "/myZnode"
byte[] bytes = zk.getData(path, false, null);
String readData = new String(bytes);

// Prints "hello world"
System.out.println(readData);

// Closing the connection
// to the ZooKeeper ensemble
zk.close();

Python3

from kazoo.client import KazooClient

# Connect to ZooKeeper
zk = KazooClient(hosts='localhost:2181')
zk.start()

# Create a node with some data
zk.ensure_path('/gfg_node')
zk.set('/gfg_node', b'some_data')

# Read the data from the node
data, stat = zk.get('/gfg_node')
print(data)

# Stop the connection to ZooKeeper
zk.stop()

Session and Watches

Session 

• Requests in a session are executed in FIFO order.
• Once the session is established then the session id is assigned to the client.
• Client sends heartbeats to keep the session valid
• session timeout is usually represented in milliseconds

Watches

• Watches are mechanisms for clients to get notifications about the changes in the Zookeeper
• Client can watch while reading a particular znode.
• Znodes changes are modifications of data associated with the znodes or changes in the znode’s children.
• Watches are triggered only once.
• If the session is expired, watches are also removed.