Retrieving File Data From HDFS using Python Snakebite

Prerequisite: Hadoop Installation, HDFS

Python Snakebite is a very popular Python library that we can use to communicate with the HDFS. Using the Python client library provided by the Snakebite package we can easily write Python code that works on HDFS. It uses protobuf messages to communicate directly with the NameNode. The python client library directly works with HDFS without making a system call to hdfs dfs. The Snakebite doesn’t support python3.  

The hdfs dfs provides multiple commands through which we can perform multiple operations on HDFS. The client library that Snakebite provides will contain various methods that allow us to retrieve data from HDFS. text() method is used to simply read the data from a file available on our HDFS. So let’s perform a quick task to understand how we can retrieve data from a file from HDFS. 

Task: Retrieving File Data From HDFS.

Step 1: Create a text file with the name data.txt and add some data to it.



cd Documents/        # Changing directory to Documents(You can choose as per your requirement)

touch data.txt         # touch command is used to create file in linux enviournment

nano data.txt        # nano is a command line text editor for Unix and Linux operating system

cat data.txt         # to see the content of a file 

Step 2: Send this data.txt file to Hadoop HDFS with the help of copyFromLocal Command.    

Syntax:

hdfs dfs -copyFromLocal /path 1 /path 2 .... /path n /destination

Using the command to sending data.txt to the root directory of HDFS.

hdfs dfs -copyFromLocal /home/dikshant/Documents/data.txt / 

Now, Check whether the file is reached to the root directory of HDFS or not with the help of the below command.

hdfs dfs -ls /

You can check it manually by visiting http://localhost:50070/ then Utilities -> Browse the file system.



Step 3: Now our task is to read the data from data.txt we send to our HDFS. So create a file data_read.py in your local file system and add the below python code to it.  

Python

filter_none

edit
close

play_arrow

link
brightness_4
code

# importing the library
from snakebite.client import Client
  
# the below line create client connection to the HDFS NameNode
client = Client('localhost', 9000)
  
# iterate over data.txt file and will show all the content of data.txt
for l in client.text(['/data.txt']):
        print l

chevron_right


Client() method explanation:

The Client() method can accept all the below listed arguments:

  • host(string): IP Address of NameNode.
  • port(int): RPC port of Namenode.
  • hadoop_version (int): Hadoop  protocol version(by default it is: 9)
  • use_trash (boolean): Use trash when removing the files.
  • effective_use (string): Effective user for the HDFS operations (default user is current user).

Step 4: Run the read_data.py file and observe the result.

python read_data.py

We have successfully fetched the data from data.txt with the help of client library.

We can also copy any file from HDFS to our Local file system with the help of Snakebite. To copy a file from HDFS create a file fetch_file.py  and copy the below python code to it. copyToLocal() method is used to achieve this.

Python



filter_none

edit
close

play_arrow

link
brightness_4
code

from snakebite.client import Client
client = Client('localhost', 9000)
for a in client.copyToLocal(['/data.txt'], '/home/dikshant/Desktop'):
        print a

chevron_right


Now, run this python file you will see the below output.

python fetch_file.py

We can observe that the file now has been copied to my /home/dikshant/desktop directory.


Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course.




My Personal Notes arrow_drop_up

Check out this Author's contributed articles.

If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.


Article Tags :
Practice Tags :


Be the First to upvote.


Please write to us at contribute@geeksforgeeks.org to report any issue with the above content.