RTF Log Forwarding to Loggly (Other External Service)

Author: Vikas Parikh

Objective

The objective of this document is to forward the logs (RTF Controller, Mule System, and Mule Applications) to Loggly. RTF 1.7 supports forwarding logs to these 4 external services Out Of the Box- 

1. Azure
2. Elastic Search
3. GELF
4. Splunk

Forwarding logs to Loggly (or other such external service), would require manual configuration- as depicted in this document.

Prerequisite

• RTF installed on the AWS as per the guide: RTF installation for AWS
• APIs are deployed to RTF as per the guide: RTF API Management
• You have subscribed to loggly.com

Configure rsyslog

• SSH into RTF controller node
• Sudo or switch to root user (sudo su) for remainder of steps
• If for whatever reason vim is not installed on the instance, run command: 

yum install vim-enhanced -y

• Ensure latest rsyslog-gnutls is installed by running command: 

yum install rsyslog-gnutls -y

• Create directory: /var/spool/rsyslog
• Edit the file /etc/rsyslog.conf and apply below 2 changes:

1. At the top of the file, uncomment the following lines under TCP description. 

This will enable TCP log traffic to be listen at port 514

#$ModLoad imtcp
#$InputTCPServerRun 514 

2. At the bottom of the file, add the following lines:

#Templates
$template remote-incoming-logs,"/var/log/runtime-fabric/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?remote-incoming-logs
& ~

This separates the application logs out into respective buckets.

Configure loggly daemon

• Download the loggly cert from the following url: 

https://logdog.loggly.com/media/logs-01.loggly.com_sha12.crt

• Import this file into the controller under the path:

/etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt

• Copy below content into a file named:  22-loggly-logger.conf and place it under: /etc/rsyslog.d/

##########################################################
##### RsyslogTemplate for Loggly ###
############################################################
#
## Setup disk assisted queues

$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1   # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g    # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on   # save messages to disk on shutdown
$ActionQueueType LinkedList    # run asynchronously
$ActionResumeRetryCount -1    # infinite retries if host is down

##RsyslogGnuTLS

$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt

template(name="LogglyFormat" type="string"
    string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-controller-{RTF_ENVIRONMENT}\"] %msg%\n"
)

template(name="MuleSystemFormat" type="string"
    string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-system-{RTF_ENVIRONMENT}\"] %msg%\n"
)

template(name="MuleApplicationFormatWarn" type="string"
    string="<12>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-application-{RTF_ENVIRONMENT}\"] %msg%\n"
)

template(name="MuleApplicationFormatErr" type="string"
    string="<11>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-application-{RTF_ENVIRONMENT}\"] %msg%\n"
)

template(name="MuleApplicationFormatInfo" type="string"
    string="<14>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-application-{RTF_ENVIRONMENT}\"] %msg%\n"
)

template(name="MuleApplicationFormatCritical" type="string"
    string="<10>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-application-{RTF_ENVIRONMENT}\"] %msg%\n"
)

# Send syslog messages to Loggly over TCP using the template.

if $programname startswith 'rsyslogd' or $programname startswith 'dhclient' or $programname startswith "run-parts" or $programname startswith 'systemd' or $programname startswith 'auditd' or $programname startswith 'anacron' or $programname startswith 'kernel' then {
    action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="LogglyFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
} else if $programname startswith 'agent' or $programname startswith 'ingress' or $programname startswith 'cluster-status' or $programname startswith "grafana" or $programname startswith "gravity-site" or $programname startswith 'influxdb' or $programname startswith 'log-forwarder' or $programname startswith 'registry-creds' or $programname startswith 'resource-cache' then {
    action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleSystemFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
} else {
    if re_match($msg, '.*ERROR.*') then {
        action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleApplicationFormatErr" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
    } else if re_match($msg,'.*WARN.*') then {
        action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleApplicationFormatWarn" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
    } else if re_match($msg, '.*CRITICAL.*') then {
        action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleApplicationFormatCritical" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
    } else {
        action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleApplicationFormatInfo" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
    }
}

• Replace {LOGGLY_TOKEN} with the customer token of your subscription. If you are not sure about customer token, you can find at: https://<subscription_domain>.loggly.com/tokens

Following command will be useful for vi editor:

:%s/{LOGGLY_TOKEN}/a7...61/

• Replace {RTF_ENVIRONMENT} with your RTF environment like dev/sit/qa/prod etc..

Following command will be useful for vi editor:

:%s/{RTF_ENVIRONMENT}/dev/

Restart the rsyslog service

• Make sure rsyslogd configuration is proper

Run the command: rsyslogd -N1

You can expect:

rsyslogd: version 8.24.0-52.el7_8.2, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

If there are scripting/configuration errors, then please revisit previous steps and resolve the errors.

• Kill process if required

Check how many rsyslog processes are running:   ps -A | grep rsyslog

You can kill specific process by: kill <process_id>

• Restart the rsyslog service: service rsyslog restart

You can expect: 

 Redirecting to /bin/systemctl restart rsyslog.service

• Check the status:  service rsyslog status

You can expect:

Redirecting to /bin/systemctl status rsyslog.service
● rsyslog.service - System Logging Service
   Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
   Active: active (running) since Fri 2020-09-18 11:55:20 UTC; 26s ago
     Docs: man:rsyslogd(8)
           http://www.rsyslog.com/doc/
 Main PID: 26445 (rsyslogd)
   CGroup: /system.slice/rsyslog.service
           └─26445 /usr/sbin/rsyslogd -n

Sep 18 11:55:20 ip-172-31-0-89.us-east-2.compute.internal systemd[1]: Starting System Logging Service...
Sep 18 11:55:20 ip-172-31-0-89.us-east-2.compute.internal rsyslogd[26445]:  [origin software="rsyslogd" swVersion="8.24.0-52.el7_8.2" x-pid="26445" x-info="ht...] start
Sep 18 11:55:20 ip-172-31-0-89.us-east-2.compute.internal rsyslogd[26445]: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.24.../2307 ]
Sep 18 11:55:20 ip-172-31-0-89.us-east-2.compute.internal systemd[1]: Started System Logging Service.
Hint: Some lines were ellipsized, use -l to show in full.

• Check rsyslog messages: sudo cat /var/log/messages | grep rsyslog

Check the messages for any errors.

Configure log-forwarder

• Enter following command: gravity enter
• Copy below content into a file named: log-forwarder.yaml and place it under new directory configs/

kind: logforwarder
version: v2
metadata:
   name: log-forwarder
spec:
   address: {CONTROLLER_IP}:514
   protocol: tcp

• The above configuration would forward gravity logs to port 514 which would be received by rsyslog and forwarded to loggly via configured loggly daemon

• Replace {CONTROLLER_IP} with your private ip of the controller 

Following command will be useful for vi editor:

:%s/{CONTROLLER_IP}/1…7/

• Run following command to apply config:

gravity resource create configs/log-forwarder.yaml

• You should expect this as response to the command:

Created log forwarder “log-forwarder”

• Enter command to exit gravity: exit

Verify the logs

• Verify that the directory /var/log/runtime-fabric/ exists
• Verify the logs exists in the directory: /var/log/runtime-fabric/
• Verify that the logs exist in the Loggly search (https://<subscription_domain>.loggly.com/search) with-  tag:mulesoft-controller-dev 
• Verify that the logs exist in the Loggly search (https://<subscription_domain>.loggly.com/search) with-  tag:mulesoft-system-dev
• Verify that the logs exist in the Loggly search (https://<subscription_domain>.loggly.com/search) with-  tag:mulesoft-application-dev