Python | Write multiple files data to master file

Given a number of input files in a source directory, write a Python program to read data from all the files and write it to a single master file.

Source directory contains n number of files, and structure is same for all files. The objective of this code is to read all the files one by one and then append the output into a single master file having structure same as source files.

Taking three input files as example, named emp_1.txt, emp_2.txt, emp_3.txt, output will contain data from all the input files.

Input: 




Output: 

 

Method #1: Using os module

filter_none

edit
close

play_arrow

link
brightness_4
code

import os
# list the files in directory
lis = os.listdir('D:\\python'
                 '\\data_files\\data_files')
print(lis)
tgt = os.listdir('D:\\python'
                 '\\data_files\\target_file')
  
file_dir ='D:\\python\\data_files\\data_files'
out_file = r'D:\\python\\data_files\\target_file\\master.txt'
ct = 0
  
print('target file :', tgt)
try:
    # check for if file exists
    # if yes delete the file 
    # otherwise data will be appended to existing file
    if len(tgt)>0:
        os.remove('D:\\python'
                  '\\data_files\\target_file\\master.txt')
  
        open(tgt, 'a').close()
    else:
        # create an empty file
        open(tgt, 'a').close()
except:
    head = open('D:\\python'
                '\\data_files\\target_file\\master.txt', 'a+')
  
    line ='empno, ename, sal'
    # write header to output
    print(head, line)
    head.close()
    # below loop to write data to output file
    for line1 in lis:
        f_dir = file_dir+'\\'+line1
        # open files in read mode
        in_file = open(f_dir, 'r+')
  
        # open output in append mode
        w = open(out_file, 'a+')
        d = in_file.readline()
        d = in_file.readlines()
        w.write("\n")
        for line2 in d:
            print(line2)
            w.write(line2)
            
        ct = ct + 1  
    w.close()      

chevron_right


Output:

 
Method #2: Using pandas

filter_none

edit
close

play_arrow

link
brightness_4
code

import pandas as pd
# pd.read_csv creates dataframes
df1 = pd.read_csv('D:\python\data_files\data_files\emp_1.txt')
df2 = pd.read_csv('D:\python\data_files\data_files\emp_2.txt')
df3 = pd.read_csv('D:\python\data_files\data_files\emp_3.txt')
  
frames = [df1, df2, df3]
  
# concat function concatenates the frames 
result = pd.concat(frames)
# to_csv function writes output to file
result.to_csv('D:\\python\\data_files'
              '\\target_file\\master.txt', encoding ='utf-8', index = False)

chevron_right


Output:



My Personal Notes arrow_drop_up

Check out this Author's contributed articles.

If you like GeeksforGeeks and would like to contribute, you can also write an article using contribute.geeksforgeeks.org or mail your article to contribute@geeksforgeeks.org. See your article appearing on the GeeksforGeeks main page and help other Geeks.

Please Improve this article if you find anything incorrect by clicking on the "Improve Article" button below.




Article Tags :

Be the First to upvote.


Please write to us at contribute@geeksforgeeks.org to report any issue with the above content.