HDF5 file stands for Hierarchical Data Format 5. It is an open-source file which comes in handy to store large amount of data. As the name suggests, it stores data in a hierarchical structure within a single file. So if we want to quickly access a particular part of the file rather than the whole file, we can easily do that using HDF5. This functionality is not seen in normal text files hence HDF5 is becoming seemingly popular in fact of being a new concept. To use HDF5, numpy needs to be imported. One important feature is that it can attach metaset to every data in the file thus provides powerful searching and accessing. Let’s get started with installing HDF5 to the computer.
To install HDF5, type this in your terminal:
pip install h5py
We will use a special tool called HDF5 Viewer to view these files graphically and to work on them. To install HDF5 Viewer, type this code :
pip install h5pyViewer
As HDF5 works on numpy, we would need numpy installed in our machine too.
python -m pip install numpy
After all the installations are done, let’s see how can we write into an HDF5 file.
Note: Working with HDF5 requires basic understanding of
numpy and its attributes, so one must be familiar with
numpy in order to understand the codes following in this article. To know more about
numpy click here.
We will create a file and save a random array of numpy in it:
In the above code, we first import the modules which were installed previously. Then we initialize the variable
arr to a random array of
numpy ranging till 1000.
Hence, we can say that this array consists of a large number of data. Next, we open the file as “write only” attribute. This means that if there isn’t any file with the name test.hdf5 then it will create one otherwise it will delete (overwrite) the content of the existing file. While opening the file, we used
with instead of
open as it has an upper-hand when compared to
open() method. We don’t need to close a file if we open it using
with Finally, we use the
.create_dataset() to set the varibale dset to the array which was created earlier.
We will now read the file which we wrote above:
Here, we open the file again but this time we open it by “read-only” attribute so that no changes can be made to the file. We set the variable
data to the data we stored in the previous file. Let’s look at the output:
It may seem that there’s nothing new. It is just an array and we are printing out the numbers just as in an array. But, the variable
data is not an array. It’s actually very different than array. It’s a dataset. Rather than storing data in the RAM, it saves it in hard-drive of the computer, thus maintaining a hierarchy of structures just like directories:
When the below line is used
data = f['default']
in the previous code, we did not access the content of the file directly but created a pointer to point at our content. Let’s look at an advantage of this:
We created two datasets but the whole procedure is same as before. A file named “test_read.hdf5” is created using the “w” attribute and it contains two datasets (array1 and array2) of random numbers. Now suppose we want to read only a selective portion of array2. For example, we want to read that part of array2 corresponding to where values of array1 are greater than 1. If we were using the conventional text files instead of HDF5 files, it would be nearly impossible to achieve this. That’s exactly were we see the power of HDF5 files:
We use the [:] to create a copy of the dataset d1 into the RAM. We did this because a dataset (the data in hard-drive) cannot be compared to the integers.
So, we conclude that HDF5 files are our best tools when we are dealing with large files as it allows us selective reading and writing of files which otherwise would have consumed a lot of memory and time.
- Downloading files from web using Python
- Working with csv files in Python
- Working with zip files in Python
- Working with PDF files in Python
- Reading and Writing to text files in Python
- Listing out directories and files in Python
- Rename multiple files using Python
- Writing files in background in Python
- Python | Write multiple files data to master file
- Upload files in Python
- Python | Reading .ini Configuration Files
- Python | Sort and store files with same extension
- Python | Move or Copy Files and Directories
- Python | Create Archives and Find Files by Name
- Python | Speech recognition on large audio files
- Python | How to use Multiple kv files in kivy
- How to move Files and Directories in Python
- Python | Merge two text files
- Python | Arrange the files in directories according to extensions
- Python Program to merge two files into a third file
If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to email@example.com. See your article appearing on the GeeksforGeeks main page and help other Geeks.
Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.