Author: Vikas Parikh
Objective
The objective of this document is to forward the logs (RTF Controller, Mule System, and Mule Applications) to Loggly. RTF 1.7 supports forwarding logs to these 4 external services Out Of the Box-
- Azure
- Elastic Search
- GELF
- Splunk
Forwarding logs to Loggly (or other such external service), would require manual configuration- as depicted in this document.
Prerequisite
- RTF installed on the AWS as per the guide: RTF installation for AWS
- APIs are deployed to RTF as per the guide: RTF API Management
- You have subscribed to loggly.com
Configure rsyslog
- SSH into RTF controller node
- Sudo or switch to root user (sudo su) for remainder of steps
- If for whatever reason vim is not installed on the instance, run command:
yum install vim-enhanced -y
- Ensure latest rsyslog-gnutls is installed by running command:
yum install rsyslog-gnutls -y
- Create directory: /var/spool/rsyslog
- Edit the file /etc/rsyslog.conf and apply below 2 changes:
- At the top of the file, uncomment the following lines under TCP description.
This will enable TCP log traffic to be listen at port 514
#$ModLoad imtcp
#$InputTCPServerRun 514
- At the bottom of the file, add the following lines:
#Templates
$template remote-incoming-logs,"/var/log/runtime-fabric/%HOSTNAME%/%PROGRAMNAME%.log"
*.* ?remote-incoming-logs
& ~
This separates the application logs out into respective buckets.
Configure loggly daemon
- Download the loggly cert from the following url:
https://logdog.loggly.com/media/logs-01.loggly.com_sha12.crt
- Import this file into the controller under the path:
/etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt
- Copy below content into a file named: 22-loggly-logger.conf and place it under: /etc/rsyslog.d/
##########################################################
##### RsyslogTemplate for Loggly ###
############################################################
#
## Setup disk assisted queues
$WorkDirectory /var/spool/rsyslog # where to place spool files
$ActionQueueFileName fwdRule1 # unique name prefix for spool files
$ActionQueueMaxDiskSpace 1g # 1gb space limit (use as much as possible)
$ActionQueueSaveOnShutdown on # save messages to disk on shutdown
$ActionQueueType LinkedList # run asynchronously
$ActionResumeRetryCount -1 # infinite retries if host is down
##RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/rsyslog.d/keys/ca.d/logs-01.loggly.com_sha12.crt
template(name="LogglyFormat" type="string"
string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-controller-{RTF_ENVIRONMENT}\"] %msg%\n"
)
template(name="MuleSystemFormat" type="string"
string="<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-system-{RTF_ENVIRONMENT}\"] %msg%\n"
)
template(name="MuleApplicationFormatWarn" type="string"
string="<12>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-application-{RTF_ENVIRONMENT}\"] %msg%\n"
)
template(name="MuleApplicationFormatErr" type="string"
string="<11>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-application-{RTF_ENVIRONMENT}\"] %msg%\n"
)
template(name="MuleApplicationFormatInfo" type="string"
string="<14>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-application-{RTF_ENVIRONMENT}\"] %msg%\n"
)
template(name="MuleApplicationFormatCritical" type="string"
string="<10>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [{LOGGLY_TOKEN}@41058 tag=\"mulesoft-application-{RTF_ENVIRONMENT}\"] %msg%\n"
)
# Send syslog messages to Loggly over TCP using the template.
if $programname startswith 'rsyslogd' or $programname startswith 'dhclient' or $programname startswith "run-parts" or $programname startswith 'systemd' or $programname startswith 'auditd' or $programname startswith 'anacron' or $programname startswith 'kernel' then {
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="LogglyFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
} else if $programname startswith 'agent' or $programname startswith 'ingress' or $programname startswith 'cluster-status' or $programname startswith "grafana" or $programname startswith "gravity-site" or $programname startswith 'influxdb' or $programname startswith 'log-forwarder' or $programname startswith 'registry-creds' or $programname startswith 'resource-cache' then {
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleSystemFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
} else {
if re_match($msg, '.*ERROR.*') then {
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleApplicationFormatErr" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
} else if re_match($msg,'.*WARN.*') then {
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleApplicationFormatWarn" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
} else if re_match($msg, '.*CRITICAL.*') then {
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleApplicationFormatCritical" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
} else {
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="MuleApplicationFormatInfo" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
}
}
- Replace {LOGGLY_TOKEN} with the customer token of your subscription. If you are not sure about customer token, you can find at: https://<subscription_domain>.loggly.com/tokens
Following command will be useful for vi editor:
:%s/{LOGGLY_TOKEN}/a7...61/
- Replace {RTF_ENVIRONMENT} with your RTF environment like dev/sit/qa/prod etc..
Following command will be useful for vi editor:
:%s/{RTF_ENVIRONMENT}/dev/
Restart the rsyslog service
- Make sure rsyslogd configuration is proper
Run the command: rsyslogd -N1
You can expect:
rsyslogd: version 8.24.0-52.el7_8.2, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.
If there are scripting/configuration errors, then please revisit previous steps and resolve the errors.
- Kill process if required
Check how many rsyslog processes are running: ps -A | grep rsyslog
You can kill specific process by: kill <process_id>
- Restart the rsyslog service: service rsyslog restart
You can expect:
Redirecting to /bin/systemctl restart rsyslog.service
- Check the status: service rsyslog status
You can expect:
Redirecting to /bin/systemctl status rsyslog.service
● rsyslog.service - System Logging Service
Loaded: loaded (/usr/lib/systemd/system/rsyslog.service; enabled; vendor preset: enabled)
Active: active (running) since Fri 2020-09-18 11:55:20 UTC; 26s ago
Docs: man:rsyslogd(8)
http://www.rsyslog.com/doc/
Main PID: 26445 (rsyslogd)
CGroup: /system.slice/rsyslog.service
└─26445 /usr/sbin/rsyslogd -n
Sep 18 11:55:20 ip-172-31-0-89.us-east-2.compute.internal systemd[1]: Starting System Logging Service...
Sep 18 11:55:20 ip-172-31-0-89.us-east-2.compute.internal rsyslogd[26445]: [origin software="rsyslogd" swVersion="8.24.0-52.el7_8.2" x-pid="26445" x-info="ht...] start
Sep 18 11:55:20 ip-172-31-0-89.us-east-2.compute.internal rsyslogd[26445]: warning: ~ action is deprecated, consider using the 'stop' statement instead [v8.24.../2307 ]
Sep 18 11:55:20 ip-172-31-0-89.us-east-2.compute.internal systemd[1]: Started System Logging Service.
Hint: Some lines were ellipsized, use -l to show in full.
- Check rsyslog messages: sudo cat /var/log/messages | grep rsyslog
Check the messages for any errors.
Configure log-forwarder
- Enter following command: gravity enter
- Copy below content into a file named: log-forwarder.yaml and place it under new directory configs/
kind: logforwarder
version: v2
metadata:
name: log-forwarder
spec:
address: {CONTROLLER_IP}:514
protocol: tcp
- The above configuration would forward gravity logs to port 514 which would be received by rsyslog and forwarded to loggly via configured loggly daemon
- Replace {CONTROLLER_IP} with your private ip of the controller
Following command will be useful for vi editor:
:%s/{CONTROLLER_IP}/1…7/
- Run following command to apply config:
gravity resource create configs/log-forwarder.yaml
- You should expect this as response to the command:
Created log forwarder “log-forwarder”
- Enter command to exit gravity: exit
Verify the logs
- Verify that the directory /var/log/runtime-fabric/ exists
- Verify the logs exists in the directory: /var/log/runtime-fabric/
- Verify that the logs exist in the Loggly search (https://<subscription_domain>.loggly.com/search) with- tag:mulesoft-controller-dev
- Verify that the logs exist in the Loggly search (https://<subscription_domain>.loggly.com/search) with- tag:mulesoft-system-dev
- Verify that the logs exist in the Loggly search (https://<subscription_domain>.loggly.com/search) with- tag:mulesoft-application-dev