How to Setup Central Logging Server with Rsyslog in Linux

This article will show us how to use Rsyslog to set up Linux as a centralized logging service. When managing logs from various systems, a centralized Rsyslog setup is advantageous. All log entries from client servers will be sent to the host server, which will allow them to be monitored and preserved in one location. Compared to sshing into each server to study its logs, this strategy makes system management considerably easier, especially if there are a lot of servers. Let’s get started with setting up the central logging server.

Prerequisite

• Minimum 2 Linux machines (Here machine we are referring to 2 instances of Linux. It can be both VM, or one VM and one Machine.)
• All should be in the network and reachable.
• rsyslog service is up and running on both machines. 

Steps for Setup Central Logging Server with Rsyslog in Linux

Step 1: Configure machine-1 as a central logging server.

By default, rsyslog uses “imjournal” and “imuxsock” modules for importing structured log messages from systemd journal and for accepting rsyslog messages from applications running on the local system via Unix sockets, respectively.

Two protocols can be used for the reception of the log messages on the server machine “TCP” and “UDP” which is using the port number “514”  by default. 

To use a TCP connection that is slower but reliable. Search and uncomment the lines below

$ModLoad imtcp
$InputTCPServerRun 514

To use a UDP connection which is faster and unreliable. search and uncomment the lines below

$ModLoad imudp
$UDPServerRun 514

We can use a port other than 514. Need to open that port from iptables and also need to turn off the firewall rule for that port. We are going to use TCP for testing purposes if we want we can UDP or both.

 Step 2: Setup the destination of the logs on the server machine.

The log messages that we will get on the server machine we need to store them at a specific logfile. For that, we need to set up the rule in rsyslog.conf file.

facility.severity_level destination

• Facility: is a type of process/application generating, they include auth, cron, daemon, kernel, local0..local7. Using “*” means all facilities.
• severity_level:  is type of log message: emerg-0, alert-1, crit-2, err-3, warn-4, notice-5, info-6, debug-7. Using “*” means all severity levels and none implies no severity level.
• destination: is either a local file or remote rsyslog server (defined in the form IP:port).

We will use the following ruleset for collecting logs from remote clients. 

$template DynamicFile,”/var/log/loghost/%fromhost-ip%.log”

if not ($fromhost-ip == ‘127.0.0.1’) then {
*.* -?DynamicFile
}

2.1: Configure rule for log destination.

We need to put this rule before all the rules in the rsyslog.conf

Now, restart the rsyslog.service to load configuration.

[root@vm-dev ~]# systemctl restart rsyslog.service

2.2: To open the port for listening we need to set rules in SELinux or Firewall whatever one is we are using.

$ sudo firewall-cmd –permanent –add-port=514/udp

$ sudo firewall-cmd –permanent –add-port=514/tcp

$ sudo firewall-cmd –reload

2.3: To check that the port is open and listening to the log using the “netstat” utility.

[root@vm-dev ~]# netstat -tulpn

It will show we that port 514 is listening on both ip4 and ipv6. 

Step 3: Configure the client machine to send logs. 

To configure the client machine we will be using the legacy method which is simple to understand and which is part of rsyslog.conf.

This will force rsyslog daemon to forward all the logs to the remote rsyslog server. As this is the legacy method. If the remote system is unreachable processing will get blocked here and discard the messages after a while. 

# Legacy method

# remote host is: name/ip:port, e.g. 192.168.0.1:514, port optional

#facility.priority @@remote-host:remote-port

*.* @@remote-host:514

# New Method

*.*  action(type=”omfwd” target=”remote-host” port=”hots-port” protocol=”tcp”

            action.resumeRetryCount=”retry-count”

            queue.type=”linkedList” queue.size=”message-queue-size”)

We are using the legacy method as it is so it will forward all the messages to the remote rsyslog server. Replace the remote host with the IP of the server machine. Just uncomment the following line and restart rsyslog service on the client machine. 

Restart the rsyslog service.

[root@vm-dev ~]# systemctl restart rsyslog.service

Step 4: Test the setup with the logger command.

We are going to test our setup with the “logger” which is provided by the rsyslog as a CLI interface for logging.

Client-Machine

[root@centos ~]# logger -t myApp -p local0.emerg “Hi, This is the test message”

Client Machine:

Server-Machine:

As we can see new log file is getting created on the server machine. The server machine can receive all the logs from the client machine. If we are facing any issues getting messages from the server please check our firewall rules. Most of the time ports are blocked from the firewall at the server as well as the client machine as well. Also, we need to check if the server machine and client machine are reachable from each other.