Reliable Messaging in Mule 4

Author: Pritam Prakash

Whenever you work in a project, you have a few connectors that are very common to all the projects. For example, FTP/SFTP, HTTP, VM, JMS are the most commonly used connectors in MuleSoft. Sometimes, in an organization, there are some applications in MuleSoft that have zero scope of message loss. That’s where the concept of reliable messaging takes its place. 

Use Case

Recently, I was working with a client whose website handled orders for subscriptions and advertisements from other companies who paid to have their services advertised on their site. The agency redirects customers to a lockbox system provided by a bank for handling payments and settlement processes. The remittance information is sent to the agency to apply for payments and settlements. Losing even a datum could could cost the agency business with its customers. 

A lockbox is a bank-operated mailing address to which a company directs its customers to send their payments. The bank opens the incoming mail, deposits all received funds in the company’s bank account, and scans the payments and any remittance information. The scanned images are posted to the SFTP folder fetched by MuleSoft to an internal file server of the agency, wherein MuleSoft converts the format of the message to excel and sends it to the company’s accounting staff where they can access the images and converted data format to clearance to outstanding accounts receivable.

Reliability patterns and message reliability approach

High reliability in messaging means that both the underlying Mule and its individual connections need to be reliable and must have zero tolerance for message loss. Any application that follows a reliability pattern follows a particular design where reliable acquisition flow couples with application logic flow and reliable messaging is ensured by built-in-support for transactions in connectors like VM, JMS and DB.

In this particular scenario:

1. We have an HTTP listener that initiates reliable acquisition flow. 
2. SFTP picks up the lockbox file from the bank’s server and passes it to the VM for transaction in the reliable acquisition flow making sure it works as a transactional item.
3. Application logic flow here is basically sub-flow that consume the file and keeps it in the agency’s server and from there picks up the file to perform the business operations. 

One scenario also includes when no file has been picked up by the SFTP connector as VM listener listens to no payload. Thus, the flow stops at publishing the message VM itself in the reliable acquisition flow. 

This also brings us to the question of where VM is used instead of flow reference to connect two flows. Well, the answer lies in the message reliability design approach itself. It’s a good design approach to implement VM or JMS even if there are tiniest possibilities to implement transactions as it makes the flow synchronous. If the reliable acquisition flow fails to deliver the message it guarantees that message is not lost in the way so it uses a VM or JMS to implement the transaction. 

If you want to move messages between different connectors that support transactions, use XA transactions to ensure that both connectors’ transactions are committed as one atomic unit. Use a try scope to enable the transactional type to ‘XA’.

Application measures to keep in mind while implementing message reliability approach:

1. Check the file repeatable stream size, and configure the streaming strategy accordingly. If you need to handle large or small files, you can change the buffer size (inMemorySize) to optimize performance.
2. File storage is only available in Mule Enterprise Edition (Mule EE). For other application use, in-memory streaming customizes the behavior by setting the initial size of the buffer (initialBufferSize), the rate at which the buffer increases (bufferSizeIncrement), the maximum buffer size (maxinMemorySize), and the unit of measurement for the buffer size value (bufferUnit).
3. VM connector uses in-memory queues by default, this makes it much faster than other transactional endpoints such as JMS.
4. Also, the inbound VM endpoint in the application logic flow should be configured as transactional to allow for rollbacks if an exception occurs during message processing.
5. Also make sure to implement the on-error propagate to implement rollback in case of such a requirement.
6. Last but not least reliability has performance implications so performance can be a factor when using too many VM’s. 

For more information on reliable messaging and patterns, you can go through the following links:

• Reliability Patterns
• Streaming in Mule Apps
• Transaction Management