Shell Scripting – System Logging

Shell scripting is a way of automating tasks and operations on a computer system by writing scripts and programs in a shell or terminal environment. These scripts can run commands, access system resources, and process data. Shell scripts are often used to automate repetitive tasks, such as backups, system maintenance, and data processing. One of the key features of shell scripting is its ability to log messages and events to the system log. This is important for a number of reasons. 

• First, it allows shell scripts to record their activities and provide a record of what has happened. This can be useful for debugging and troubleshooting issues with the script.
• Second, logging can help to identify patterns and trends in system usage and performance. By analyzing log messages, it is possible to identify areas where the system is performing poorly or where there are opportunities for improvement.

There are several tools and frameworks available for logging in shell scripts. The syslog protocol is a widely-used logging standard that allows programs and processes to send log messages to a central logging server or service. Other tools, such as the logger command, can be used to send log messages directly to the system log. In addition to logging their own activities, shell scripts can also capture and propagate log messages from other programs and processes. This can be useful for aggregating log messages from different sources and making them easier to analyze and troubleshoot.

Overall, logging is an important feature of shell scripting, as it allows scripts to record their activities and capture log messages from other programs and processes, providing a valuable source of information for debugging, analysis, and system improvement.

Syslog

The syslog is a standardized logging system that is used on many computer systems and network devices to log system events and messages. It provides a central repository for log messages, allowing administrators to easily collect and review important information about their systems.

The syslog can be used to log a wide range of system events, including system startup and shutdown, authentication and security events, system messages, and other important events. This can be useful for monitoring system activity, diagnosing problems, and identifying potential security threats.

One of the key benefits of the syslog is its ability to collect log messages from a wide range of sources, including servers, applications, and network devices. This allows administrators to view log messages from multiple sources in a single location, making it easier to identify patterns and trends in system activity.

The syslog also provides a number of options for filtering and parsing log messages, allowing administrators to focus on specific types of events or messages. It also supports different levels of severity, allowing administrators to prioritize important messages and focus on critical issues.

Overall, the syslog is a powerful tool for managing and monitoring systems and networks, providing a central location for collecting and reviewing important log messages and helping administrators to identify and troubleshoot problems and security threats.

How to use syslog in shell scripting

The “syslog” command or utility is a tool that can be used to send log messages to the system log shell scripts. It is available on most Unix-like operating systems, such as Linux, macOS, and BSD.

To use the syslog command in a shell script, you can simply include it in the script along with the desired options and arguments. For example, the following command sends a log message with the “error” severity level to the system log:

syslog -s -p “error” “This is an error message”

The “-s” option specifies that the message should be sent to the system log, and the “-p” option specifies the severity level of the message. Other options are available for controlling the format and destination of the log message.

Here are a few examples of using the syslog command in shell scripts:

Logging a message with the “info” severity level:

syslog -s -p “info” “This is an informational message”

Logging a message with the “warning” severity level:

syslog -s -p “warning” “This is a warning message”

Logging a message with the “error” severity level:
syslog -s -p “error” “This is an error message”

Logging a message with a custom tag:
syslog -s -t “custom_tag” “This is a message with a custom tag”

Logging a message with a custom facility:
syslog -s -f “local0” “This is a message with a custom facility”

Logger

The logger command is a command-line utility that is used to send messages to the syslog on a Unix-like system. It allows you to easily log information to the syslog from the command line or from within a script, without the need to use the syslog command or library directly.

To use the logger command, you simply specify the message you want to log as an argument to the command. For example, the following command sends a message with the “info” severity level to the syslog:

logger "This is an informational message"

The logger command supports a variety of options that allow you to control the severity level, facility, and other aspects of the log message. For example, the following command sends a message with the “warning” severity level and the “local0” facility:

logger -p “warning” -f “local0” “This is a warning message”

Using the logger command in shell scripts allows you to easily log messages to the syslog, providing a record of the script’s activities and any important messages or events that it generates. This can be useful for debugging and troubleshooting issues with the script, as well as for monitoring system activity and performance.

Examples

Here is an example of a shell script that uses the logger command to iterate over directories, check all files, and validate any CSV files it finds:

#!/bin/bash
# Iterate over all directories in the current directory
for dir in ./*/
do
 
 # Change to the directory
 cd "$dir"
 
 # Iterate over all files in the directory
 for file in *
 do
   
   # Check if the file is a CSV
   if [ ${file##*.} == "csv" ]; then
     
     # Validate the CSV file
     csv_valid=$(python -c "import csv; from io import StringIO; \
       try: csv.reader(StringIO(open('$file', 'r').read())); \
       except Exception as e: print(e)")
     
     # Log the validation result to the syslog
     logger -t "csv_validator" "Validated $file - $csv_valid"
   fi
 done
 
 # Go back to the parent directory
 cd ..
done
 

In this script, the outer for loop iterates over all directories in the current directory. For each directory, the script changes to that directory and then iterates over all files in the directory. If a file is a CSV file, the script uses the python command to validate the file and logs the result to the syslog using the logger command. This allows the administrator to easily monitor the script’s progress and check the validation results for each CSV file.

Code Explanation

for dir in ./*/
do
# Change to the directory
cd "$dir"

# Iterate over all files in the directory

This code will iterate over all the directories inside the current directory.

for file in *
do
 
  # Check if the file is a CSV
  if [ ${file##*.} == "csv" ]; then
    # Validate the CSV file

This line of code will iterate over all the files in the directory the if condition checks if the file is a CSV file. if it is not a CSV file it will skip the same file.

csv_valid=$(python -c “import csv; from io import StringIO; \

      try: csv.reader(StringIO(open(‘$file’, ‘r’).read())); excep Exception as e:  \

      “)

    # Log the validation result to the syslog

    logger -t “csv_validator” “Validated $file – $csv_valid”

This line of code will open the CSV file using the python CSV module and check if the same is valid result will be saved in the variable csv_valid. if the CSV is valid the next line will log it using the logger below:

Dec 09 22:36:07 hostname csv_validator: Validated file1.csv – None

if the CSV file is not valid the logger will log as below

Dec 09 22:36:10 hostname csv_validator: Validated file4.csv – line contains NULL byte

Where the end of the line contains information about the error like this

line contains NULL byte

How to Check the output of the logger

To check the output file for the logger command, you can use the tail command to display the last few lines of the file. By default, the logger command logs messages to the /var/log/syslog file on most Unix-like systems. You can use the following command to view the last few lines of this file:

tail /var/log/syslog 

This command will display the last few lines of the /var/log/messages file, which should include any messages that were logged by the logger command. You can also use the grep command to search for specific log messages, using the -i flag to perform a case-insensitive search:

tail /var/log/syslog  | grep -i “csv_validator”

This command will display only the log messages that were logged by the csv_validator script, making it easier to see the results of the CSV validation process. You can also use the -n flag with the tail command to specify the number of lines you want to display, and the -f flag to follow the log file in real-time as new messages are added. This can be useful for monitoring the log file as your script runs.

Sample output

Dec 09 22:36:07 hostname csv_validator: Validated file1.csv – None

Dec 09 22:36:08 hostname csv_validator: Validated file2.csv – None

Dec 09 22:36:09 hostname csv_validator: Validated file3.csv – None

Dec 09 22:36:10 hostname csv_validator: Validated file4.csv – line contains NULL byte

Dec 09 22:36:11 hostname csv_validator: Validated file5.csv – expected 2 fields, saw 3

Dec 09 22:36:12 hostname csv_validator: Validated file6.csv – None

Dec 09 22:36:13 hostname csv_validator: Validated file7.csv – None

Dec 09 22:36:14 hostname csv_validator: Validated file8.csv – None

Dec 09 22:36:15 hostname csv_validator: Validated file9.csv – line contains NULL byte

Dec 09 22:36:16 hostname csv_validator: Validated file10.csv – None

In these examples, the script has validated 10 CSV files and logged the results to the Syslog. The log entries show the name of each file that was validated, along with the result of the validation. In some cases, the validation was successful and the log entry shows “None” for the result. In other cases, the validation failed and the log entry shows the reason for the failure.

Actual output