rsyslog Forwarding

Collect and Redirect Logs to a Remote Server

Compiling all of your machine's application logs in one location will be helpful for analyzing at a later time.
Now that you have gathered all of the logs in one location, you can begin processing them. What happens, though, if your computer is merely one node among several servers? In this instance, local log processing provides some but not all of the information you need to understand this particular node.
At this point, you might wish to move all of the collected logs to a central server, which will process the data, look for problems and inconsistencies, set off alarms, and then store the logs for further forensic examination.

It is convenient to have all of your logs accessible from one location. It is not necessary to constantly switch between machines, looking for relevant data and manually superimposing several log files.
In order to accomplish the aforementioned, you can take advantage of the syslog protocol and a well-known syslog daemon named rsyslog to gather all the logs and send them securely and dependably to a distant server for additional processing.
And that's precisely the example I want to give in this lesson to highlight a typical and significant Syslog use case. I will introduce it to others who do not know it and let them sample some of the issues it can resolve.

Syslog protocol

A standard for message recording is the system logging protocol, or Syslog. It establishes a standard message format so that computers, whether connected locally or remotely, can transmit messages to a logging server. The messages will then be processed or transported further by the logging server.

All you need to do is forward your messages to a logging server or, alternatively, a logging daemon, and forget about them, provided that their format complies with this protocol.

From then on, the logging server and the infrastructure it connects to will manage the message transit, rotation, processing, and enrichment. None of that information or topic needs to be included in your application. As a result, we have a decoupled design, where the application and log processing are distinct.

But uniformity is, of course, the primary goal of the Syslog protocol. First of all, when all the programs follow a common standard that produces logs in the same format—or at least nearly the same—parsing the logs becomes considerably easier. But let's not jump the gun just yet.

First of all, it's simple to filter the records by a certain time window or by the corresponding log levels—also known as severity levels—if your logs follow a standard format. For instance: mistake, warning, information, and so forth.

Second, you might have numerous applications that each implement a logtransport in their own right. If that were the case, setting up file logging, log rotation, or log forwarding for each application would require you to spend a significant amount of time reading through the documentation rather than just setting it up once in your syslog server and expecting all of your apps to automatically submit their logs to it.

Log Levels

Syslog supports the following log levels referred to as severity levels as per syslogs’s terminology:

These levels allow you to categorize messages by the severity (importance) criteria, with emergency being the highest level.

Configure rsyslog

As rsyslog typically comes preinstalled on most common Linux distros, make sure your rsylogd is updated.

Run the following command across all your servers:

rsyslogd -v

And ensure that the version in the output is 6 or higher.

For Ubuntu:

sudo apt-get update
sudo apt-get install --only-upgrade rsyslog
sudo systemctl restart rsyslog
sudo apt-get install rsyslog-gnutls
sudo systemctl restart rsyslog

For RedHat:

sudo yum update rsyslog
sudo systemctl restart rsyslog
sudo yum install epel-release
sudo yum install rsyslog-gnutls
sudo systemctl restart rsyslog

Configure rsyslog server

For the nodes that will be exporting their logs to the central server, we will now build an rsyslog configuration file. Create the configuration file in the rsyslog config directory to accomplish this:

sudo touch /etc/rsyslog.d/export-syslog.conf
Ensure that the file is readable by the syslog user on Ubuntu
chown syslog:adm /etc/rsyslog.d/export-syslog.conf
chmod 644 /etc/rsyslog.d/export-syslog.conf
vi /etc/rsyslog.d/export-syslog.conf

# Set certificate files
global(
  DefaultNetstreamDriverCAFile="<path_to_your_ca.pem>"
  DefaultNetstreamDriverCertFile="<path_to_your_cert.pem>"
  DefaultNetstreamDriverKeyFile="<path_to_your_private_key.pem>"
)

# Set up the forwarding action for all messages
*.* action(
  type="omfwd"
  StreamDriver="gtls"
  StreamDriverMode="1"
  StreamDriverPermittedPeers="<domain_name_of_your_accepting_central_server>"
  StreamDriverAuthMode="x509/name"
  target="<domain_name_or_ip_of_your_accepting_central_server>" port="514" protocol="tcp"
  action.resumeRetryCount="100" # you may change the queue and retry params as you see fit
  queue.type="linkedList" queue.size="10000"
)

Configure accepting rsyslog server

Now let's set up a central server to receive the logs from the remaining computers.
Now, make an rsyslog configuration file in the same manner that you did for the exporting server:

sudo touch /etc/rsyslog.d/import-syslog.conf
vi /etc/rsyslog.d/import-syslog.conf

# Set certificate files
global(  DefaultNetstreamDriverCAFile="/etc/ssl/certs/<your_domain_name>-letsencrypt-fullchain.pem"
  DefaultNetstreamDriverCertFile="/etc/ssl/certs/<your_domain_name>-letsencrypt-cert.pem"
  DefaultNetstreamDriverKeyFile="/etc/ssl/private/<your_domain_name>-letsencrypt-privkey.pem"
)

# TCP listener
module(
  load="imtcp"
  PermittedPeer=["<your_peer1>","<your_peer2>","<your_peer3>"]
  StreamDriver.AuthMode="x509/name"
  StreamDriver.Mode="1"
  StreamDriver.Name="gtls"
)

# Start up listener at port 514
input(
  type="imtcp"
  port="514"
)

Restart rsyslog

You now need to restart rsyslog on each server you've provided the necessary configurations to, starting with the central accepting node:

sudo systemctl restart rsyslog
You can check if there are any errors after the rsyslog restart by executing the following:

sudo systemctl status rsyslog
sudo journalctl -u rsyslog | tail -100

The 100 final lines of rsyslog's logs will be printed by the second command above, while the first will show the current state of rsyslog. You should discover useful information there if your setup failed due to a misconfiguration.

Forwarding to More than One Server

If you want to forward to more than one server, you can accomplish it fast. You can define as many targets as you'd like because Rsyslog doesn't have any restrictions on the quantity or kind of actions. But it's crucial to understand that an action consists of all of the instructions. Thus, you must duplicate the rule configuration in addition to adding (only) a second forwarding rule. Use caution when naming the queue files for the second action to avoid messing up your system.

$ModLoad imuxsock # local message reception
$WorkDirectory /rsyslog/work # default location for work (spool) files

# start forwarding rule 1
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd1 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server1:port
# end forwarding rule 1

# start forwarding rule 2
$ActionQueueType LinkedList # use asynchronous processing
$ActionQueueFileName srvrfwd2 # set file name, also enables disk mode
$ActionResumeRetryCount -1 # infinite retries on insert failure
$ActionQueueSaveOnShutdown on # save in-memory data if rsyslog shuts down
*.* @@server2
# end forwarding rule 2

Take note that the filenames srvrfwd1 and srvrfwd2 were used for the first and second rules, respectively. In the second forwarding rule, server did not have a port name. There, you can additionally designate a port or remove the port from server1.

When there are several action queues, each one functions separately. So, even in the event that server 1 goes down, server 2 continues to receive data instantly. The client won't block and will hold off till server 1 restarts. Likewise, the state of server2 won't have an impact on server1's functionality.

Ref: https://www.freecodecamp.org/news/what-is-syslog-handbook/