Author: Jitendra Bafna

Introduction

Logging is a very important part of your environment for visualizing the data, troubleshooting, debugging and to identify the production issues quickly.

Logging must be robust, consistent and reliable and most importantly that we can have some centralized and easy to use tools which can be used for logging purposes. There are various logging tools like Splunk, ELK etc.

MuleSoft provides its own logging framework for storing application and system logs. Main reason for using a custom logging framework is that MuleSoft in cloudhub stores data for 30 days or upto 100 MB. 

Once the application is deleted, we will lose all the logs. So we required logging tools like Splunk for storing data and can persist for a longer time.

Splunk logging tool can be used to enable log MuleSoft On Premise Runtime logs, On Premise MuleSoft Application and CloudHub Application logs.

In the last article, we have seen how to set up Splunk and enable logging for MuleSoft Application. Click herefor more details.

Enable Custom Logging For CloudHub Application

Before enabling the logging for cloudhub application, you need Disable CloudHub logs. By default this option is not available and you need to raise ticket with MuleSoft for providing this option. 

Once you disabled cloudhub logs, MuleSoft is not responsible for below things

  • MuleSoft is not responsible for lost logging data due to misconfiguration of your own log4j appender.
  • MuleSoft is also not responsible for misconfigurations that result in performance degradation, running out of disk space, or other side effects.
  • When you disable the default CloudHub application logs, then only the system logs are available. For application worker logs, please check your own application’s logging system. Downloading logs is not an option in this scenario.
  • Only Asynchronous log appenders can be used, Synchronous appenders should not be used.
  • Use asynchronous loggers and not synchronous ones is to avoid threading issues. Synchronous loggers can lock threads waiting for responses.

You need to create log4j2.xml at location src/main/resources.

There are 2 ways you can send logs to Splunk 1.) HTTP Appender 2.) Splunk HTTP

We will see how we can send logs from cloudhub application to Splunk using Splunk HTTP

You will be requiring the token and connection details related to Splunk. So you can go through the article and understand how to fetch those details.

You need to add a SplunkHttp appender inlog4j.xml. Provide url to connect Splunk, source, pattern layout etc.

<SplunkHttp name="SPLUNK-CLOUD" source="mule-cloudhub-app"
			url="http://localhost:8088/" 
			token="b3a34eabd-b5f4-471431-97d2-e5b234af5bae9e" index="main"
			disableCertificateValidation="true">
			<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</SplunkHttp>

You need to make sure the logger must be asynchronous and it is recommended by MuleSoft also.

While using SplunkHttp, you need to add the following dependency in pom.xml of your application.

<dependency>
	<groupId>com.splunk.logging</groupId>
	<artifactId>splunk-library-javalogging</artifactId>
	<version>1.7.3</version>
</dependency>

You may require to add below two additional dependencies.

<dependency>
	<groupId>org.apache.logging.log4j</groupId>
	<artifactId>log4j-core</artifactId>
	<version>2.10.0</version>
</dependency>
<dependency>
	<groupId>org.apache.logging.log4j</groupId>
	<artifactId>log4j-api</artifactId>
	<version>2.10.0</version>
</dependency>

You will be also required to add a repository into your pom.xml of your application.

<repository>
	<id>splunk-artifactory</id>
	<name>Splunk Releases</name>
	<url>http://splunk.jfrog.io/splunk/ext-releases-local</url>
</repository>

You can add more loggers like Log4J2CloudhubLogAppender into your log4j2.xml to enable logging on the cloudhub log console of your application.

<Log4J2CloudhubLogAppender name="CLOUDHUB"
			addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
			applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
			appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
			appendMaxAttempts="${sys:logging.appendMaxAttempts}"
			batchSendIntervalMs="${sys:logging.batchSendInterval}"
			batchMaxRecords="${sys:logging.batchMaxRecords}" memBufferMaxSize="${sys:logging.memBufferMaxSize}"
			journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
			journalMaxFileSize="${sys:logging.journalMaxFileSize}"
			clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
			clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
			clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
			serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
			serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
			statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
			<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
</Log4J2CloudhubLogAppender>

Below is full log4j2.xml which can be used for your application for enabling custom logging on cloudhub and splunk.

<?xml version="1.0" encoding="UTF-8"?>
<Configuration status="INFO" name="cloudhub"
	packages="com.mulesoft.ch.logging.appender,com.splunk.logging,org.apache.logging.log4j">
	<Appenders>
		<Log4J2CloudhubLogAppender name="CLOUDHUB"
			addressProvider="com.mulesoft.ch.logging.DefaultAggregatorAddressProvider"
			applicationContext="com.mulesoft.ch.logging.DefaultApplicationContext"
			appendRetryIntervalMs="${sys:logging.appendRetryInterval}"
			appendMaxAttempts="${sys:logging.appendMaxAttempts}"
			batchSendIntervalMs="${sys:logging.batchSendInterval}"
			batchMaxRecords="${sys:logging.batchMaxRecords}" memBufferMaxSize="${sys:logging.memBufferMaxSize}"
			journalMaxWriteBatchSize="${sys:logging.journalMaxBatchSize}"
			journalMaxFileSize="${sys:logging.journalMaxFileSize}"
			clientMaxPacketSize="${sys:logging.clientMaxPacketSize}"
			clientConnectTimeoutMs="${sys:logging.clientConnectTimeout}"
			clientSocketTimeoutMs="${sys:logging.clientSocketTimeout}"
			serverAddressPollIntervalMs="${sys:logging.serverAddressPollInterval}"
			serverHeartbeatSendIntervalMs="${sys:logging.serverHeartbeatSendIntervalMs}"
			statisticsPrintIntervalMs="${sys:logging.statisticsPrintIntervalMs}">
			<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
		</Log4J2CloudhubLogAppender>
		<SplunkHttp name="SPLUNK-CLOUD" source="mule-cloudhub-app"
			url="http://localhost:8088/" 
			token="b3a34eabd-b5f4-471431-97d2-e5b234af5bae9e" index="main"
			disableCertificateValidation="true">
			<PatternLayout pattern="[%d{MM-dd HH:mm:ss}] %-5p %c{1} [%t]: %m%n" />
		</SplunkHttp>
	</Appenders>
	<Loggers>
		<AsyncLogger
			name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor"
			level="INFO" />
		<AsyncLogger name="com.mulesoft.agent" level="INFO" />
		<AsyncRoot level="INFO">
	                           <AppenderRef ref="SPLUNK-CLOUD" />
                                            <AppenderRef ref="CLOUDHUB" />
		</AsyncRoot>
		<AsyncLogger name="com.gigaspaces" level="ERROR" />
		<AsyncLogger name="com.j_spaces" level="ERROR" />
		<AsyncLogger name="com.sun.jini" level="ERROR" />
		<AsyncLogger name="net.jini" level="ERROR" />
		<AsyncLogger name="org.apache" level="WARN" />
		<AsyncLogger name="org.apache.cxf" level="WARN" />
		<AsyncLogger name="org.springframework.beans.factory"
			level="WARN" />
		<AsyncLogger name="org.mule" level="INFO" />
		<AsyncLogger name="com.mulesoft" level="INFO" />
		<AsyncLogger name="org.jetel" level="WARN" />
		<AsyncLogger name="Tracking" level="WARN" />
		<AsyncLogger name="org.mule" level="INFO" />
		<AsyncLogger name="com.mulesoft" level="INFO" />
		<AsyncLogger name="org.mule.extensions.jms" level="INFO" />
		<AsyncLogger name="org.mule.service.http.impl.service.HttpMessageLogger"
			level="INFO" />
		<AsyncLogger name="org.mule.extension.salesforce" level="INFO" />
		<AsyncLogger name="org.mule.extension.ftp" level="INFO" />
		<AsyncLogger name="org.mule.extension.sftp" level="INFO" />
		<AsyncLogger name="com.mulesoft.extension.ftps" level="INFO" />
		<AsyncLogger name="org.mule.modules.sap" level="INFO" />
		<AsyncLogger name="com.mulesoft.extension.mq" level="INFO" />
		<AsyncLogger name="com.mulesoft.mq" level="INFO" />
		<AsyncLogger name="org.mule.extension.db" level="INFO" />
		<AsyncLogger name="httpclient.wire" level="DEBUG" />
		<AsyncLogger name="org.mule.transport.email" level="DEBUG" />
	</Loggers>
</Configuration>

JSON Logger

JSON Logger is a component for logging the information, warning, errors. It logs the data in the form of JSON. It is recommended to use JSON Logger for your application and it logs the data in a well formed way in any tools like Splunk, ELK etc.

You can install JSON Logger in Anypoint Studio from Exchange.

Configure the JSON Logger as shown below.

You can set the application name, application version and environment in properties file.

<?xml version="1.0" encoding="UTF-8"?>

<mule xmlns:ee="http://www.mulesoft.org/schema/mule/ee/core"
	xmlns:json-logger="http://www.mulesoft.org/schema/mule/json-logger"
	xmlns:api-gateway="http://www.mulesoft.org/schema/mule/api-gateway"
	xmlns:http="http://www.mulesoft.org/schema/mule/http" xmlns="http://www.mulesoft.org/schema/mule/core" xmlns:doc="http://www.mulesoft.org/schema/mule/documentation" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans-current.xsd 
	http://www.mulesoft.org/schema/mule/core http://www.mulesoft.org/schema/mule/core/current/mule.xsd
	http://www.mulesoft.org/schema/mule/http http://www.mulesoft.org/schema/mule/http/current/mule-http.xsd 
http://www.mulesoft.org/schema/mule/api-gateway http://www.mulesoft.org/schema/mule/api-gateway/current/mule-api-gateway.xsd
http://www.mulesoft.org/schema/mule/json-logger http://www.mulesoft.org/schema/mule/json-logger/current/mule-json-logger.xsd
http://www.mulesoft.org/schema/mule/ee/core http://www.mulesoft.org/schema/mule/ee/core/current/mule-ee.xsd">
	<http:request-config name="HTTP_Request_configuration" doc:name="HTTP Request configuration" doc:id="4dea45bf-d0c6-4ee8-baa0-0b8a9e508c88" basePath="/api/timezone" >
		<http:request-connection host="worldtimeapi.org" />
	</http:request-config>
	<configuration-properties doc:name="Configuration properties" doc:id="9dacbbe5-0ec7-4ce4-b094-ff89131a07cc" file="cloud.properties" />

	<global-property doc:name="Global Property" doc:id="7a60a2c8-efcf-4538-81c8-64248fe0dec4" name="mule.env" value="dev" />
	<http:listener-config name="HTTP_Listener_config" doc:name="HTTP Listener config" doc:id="694e8fba-882a-4f04-805b-50c66ecec1a3" >
		<http:listener-connection host="0.0.0.0" port="${http.private.port}" />
	</http:listener-config>
	<json-logger:config name="JSON_Logger_Config" doc:name="JSON Logger Config" doc:id="59e12b0a-d64d-4300-af40-23b29f11c2dc"/>
	<flow name="test-application-cloudFlow" doc:id="7db040d5-7309-4688-a4c5-fea0f43c6103">
		<http:listener doc:name="Listener" doc:id="da11b76c-b4d5-4140-a539-5841c4690b8c" config-ref="HTTP_Listener_config" path="/v1.0/timezone/{region}/{city}">
			<http:error-response statusCode="#[vars.httpStatusCode]" >
				<http:body ><![CDATA[#[payload]]]></http:body>
			</http:error-response>
		</http:listener>
		<http:request method="GET" doc:name="Request" doc:id="730aec0d-fb82-4430-8cf3-c14264a7e9ae" config-ref="HTTP_Request_configuration" path="/{region}/{city}">
			<http:uri-params ><![CDATA[#[output application/java
---
{
	city : attributes.uriParams.city,
	region : attributes.uriParams.region
}]]]></http:uri-params>
		</http:request>
		<json-logger:logger doc:name="Logger" doc:id="f2257ace-a8b0-4f5f-83d5-4ac8a710983f" config-ref="JSON_Logger_Config" message="#['Information']"/>
		<error-handler >
			<on-error-propagate enableNotifications="true" logException="true" doc:name="On Error Propagate" doc:id="1d5f427f-ceaa-4af1-8e2d-2be0aaf45b46" type="HTTP:FORBIDDEN">
				<ee:transform doc:name="Transform Message" doc:id="b978dae0-7538-4910-82ee-3c12236b6603" >
					<ee:message >
						<ee:set-payload ><![CDATA[%dw 2.0
output application/json
---
{
	error:error.description
}]]></ee:set-payload>
					</ee:message>
					<ee:variables >
						<ee:set-variable variableName="httpStatusCode" ><![CDATA[403]]></ee:set-variable>
					</ee:variables>
				</ee:transform>
				<json-logger:logger doc:name="Logger" doc:id="3c5e26da-4183-4318-9f6a-a07b86b717ff" config-ref="JSON_Logger_Config" message="Data From Timezone" priority="ERROR"/>
			</on-error-propagate>
		</error-handler>
	</flow>
</mule>

Now, you can verify the logs in Splunk and it will be in JSON format.

Enable Splunk Logging For On Premise MuleSoft Runtime 

For enabling the logging for On Premise MuleSoft Runtime, you can use HTTP Appender.

You can add HTTP Appender in log4j2.xml located at $MULE_HOME/conf/

<Http name="Splunk" url="http://localhost:8088/services/collector/raw">
			<Property name="Authorization" value="Splunk a532e102-5f55-40ad-a047-b55a822972332"></Property>
<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n"></PatternLayout>
</Http>

This will allows to send logs to Splunk.

Full log4j2.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration>
	<Appenders>
		<Console name="Console" target="SYSTEM_OUT">
			<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n" />
		</Console>
		<Console name="ConsoleLogUtil" target="SYSTEM_OUT">
			<PatternLayout pattern="%m%n" />
		</Console>
		<Http name="Splunk" url="http://localhost:8088/services/collector/raw">
			<Property name="Authorization" value="Splunk a532e102-5f55-40ad-a047-b55a822972332"></Property>
			<PatternLayout pattern="%-5p %d [%t] [event: %X{correlationId}] %c: %m%n"></PatternLayout>
		</Http>
	</Appenders>
	<Loggers>
		<AsyncLogger
			name="org.mule.runtime.core.internal.processor.LoggerMessageProcessor"
			level="INFO" />
		<AsyncLogger name="com.mulesoft.agent" level="INFO" />
		<AsyncLogger name="org.mule.runtime.core.internal.logging"
			additivity="false" level="INFO">
			<AppenderRef ref="ConsoleLogUtil" />
		</AsyncLogger>
		<AsyncRoot level="INFO">
			<AppenderRef ref="Console" />
			<AppenderRef ref="Splunk" />
		</AsyncRoot>
		<AsyncLogger name="com.mulesoft.mule.runtime.gw" level="INFO" />
		<AsyncLogger name="com.mulesoft.extension.policies" level="INFO" />
		<AsyncLogger name="com.mulesoft.extension.gateway" level="INFO" />
		<AsyncLogger name="com.mulesoft.analytics" level="INFO" />
		<AsyncLogger name="org.mule" level="INFO" />
		<AsyncLogger name="com.mulesoft" level="INFO" />
	</Loggers>
</Configuration>

This is how you can enable Splunk logging using SplunkHTTP and HTTP Appender for MuleSoft applications.

Leave a Comment

You must be logged in to post a comment.