Skip to content
Related Articles

Related Articles

Python | Write multiple files data to master file
  • Last Updated : 06 May, 2019

Given a number of input files in a source directory, write a Python program to read data from all the files and write it to a single master file.

Source directory contains n number of files, and structure is same for all files. The objective of this code is to read all the files one by one and then append the output into a single master file having structure same as source files.

Taking three input files as example, named emp_1.txt, emp_2.txt, emp_3.txt, output will contain data from all the input files.

Input: 




Output: 

 

Method #1: Using os module



filter_none

edit
close

play_arrow

link
brightness_4
code

import os
# list the files in directory
lis = os.listdir('D:\\python'
                 '\\data_files\\data_files')
print(lis)
tgt = os.listdir('D:\\python'
                 '\\data_files\\target_file')
  
file_dir ='D:\\python\\data_files\\data_files'
out_file = r'D:\\python\\data_files\\target_file\\master.txt'
ct = 0
  
print('target file :', tgt)
try:
    # check for if file exists
    # if yes delete the file 
    # otherwise data will be appended to existing file
    if len(tgt)>0:
        os.remove('D:\\python'
                  '\\data_files\\target_file\\master.txt')
  
        open(tgt, 'a').close()
    else:
        # create an empty file
        open(tgt, 'a').close()
except:
    head = open('D:\\python'
                '\\data_files\\target_file\\master.txt', 'a+')
  
    line ='empno, ename, sal'
    # write header to output
    print(head, line)
    head.close()
    # below loop to write data to output file
    for line1 in lis:
        f_dir = file_dir+'\\'+line1
        # open files in read mode
        in_file = open(f_dir, 'r+')
  
        # open output in append mode
        w = open(out_file, 'a+')
        d = in_file.readline()
        d = in_file.readlines()
        w.write("\n")
        for line2 in d:
            print(line2)
            w.write(line2)
            
        ct = ct + 1  
    w.close()      

chevron_right


Output:

 
Method #2: Using pandas

filter_none

edit
close

play_arrow

link
brightness_4
code

import pandas as pd
# pd.read_csv creates dataframes
df1 = pd.read_csv('D:\python\data_files\data_files\emp_1.txt')
df2 = pd.read_csv('D:\python\data_files\data_files\emp_2.txt')
df3 = pd.read_csv('D:\python\data_files\data_files\emp_3.txt')
  
frames = [df1, df2, df3]
  
# concat function concatenates the frames 
result = pd.concat(frames)
# to_csv function writes output to file
result.to_csv('D:\\python\\data_files'
              '\\target_file\\master.txt', encoding ='utf-8', index = False)

chevron_right


Output:

Attention geek! Strengthen your foundations with the Python Programming Foundation Course and learn the basics.

To begin with, your interview preparations Enhance your Data Structures concepts with the Python DS Course.

My Personal Notes arrow_drop_up
Recommended Articles
Page :